Is that a manhole cover or dragonfly sitting on a table? Is that a green iguana or just a squirrel running with some nuts? Is that a unicycle or a crocodile crossing the road? To humans, the answer is obvious. But the best image-identifying artificial intelligence in the world hasn’t a clue.

That’s because each of these images was carefully selected to fool state-of-the-art image recognition technology. They’re part of a collection of 7,000 images curated by researchers from UC Berkeley, the University of Washington, and the University of Chicago.

“Current [machine learning] models are brittle,” says Dan Hendrycks, a PhD student in computer science at UC Berkeley who was an author on the paper. “While other research uses artificial data to study robustness, we show that models make egregious and highly consistent mistakes on real data [with real photos].”

Unicycle [Image: courtesy Dan Hendrycks]

To understand why that matters, let’s rewind. Over the past few years, image recognition has gotten really good, really fast. That’s largely thanks to an ever-growing open data set created by the University of Stanford called ImageNet. The collection now consists of over 14 million photos, which are each labeled with identifiers like “tree” and “sky.” This massive database is a training set, or a reference for new AI systems to learn how to identify images, much like a toddler can reference a picture book to slowly learn new words. Artificial intelligence trained with ImageNet—which you’d probably know best from Microsoft services like Bing—have gotten extremely accurate, able to identify objects with accuracy as high as 95%. This is actually better than humans performing the same job!

But closing that last 5% accuracy gap is an extremely large problem. Since 2017, computers haven’t been getting more accurate at identifying images. That’s why researchers are exploring ways to understand those few images that computers can’t seem to parse. The team behind the new collection scoured Flickr by hand, looking for photos they thought might confuse software. They’d test them against AI models trained on ImageNet, and if the images proved confusing, these photos were added to their new data set—which they dubbed ImageNet-A. It’s basically the anti-ImageNet. The 7,000 photos in this collection drop the accuracy of AI from over 90% to a mere 2%. Yes, you read that right. Ninety-eight times out of 100, the best vision AI models in the world will be confused by these photos.

The question of why AI systems don’t understand these images is complex. Teaching AI today tends to involve throwing a lot of data into a black box—in other words, you can only judge its accuracy based on its final conclusion, not the process it took to get there. If that black box sees enough variations of a tree that it begins identifying new trees in new photos, we consider it successful. (This repetitious task is known as machine learning.) The problem is, we don’t know why the AI has decided that a tree is a tree. Is it the shape? Color? Context? Texture? Is it because trees have some unifying core geometry that humans have never recognized? We don’t know. AI is judged by its answers, not its reasoning. That means we can get all sorts of unexpected bias from AI, which poses a major problem when AI systems are being used in technology like autonomous cars or fields like criminal justice. It also means that image recognition systems aren’t intelligent in any real way; they’re more like match game savants.

Sea Lion [Image: courtesy Dan Hendrycks]

Building ImageNet-A is about tricking AI to discover why certain images confuse these systems. For example, when an AI mistakes one of the images—of a squirrel—for a sea lion, the lack of deeper intelligence and reasoning starts to become clear. The system is relying on texture of these animals alone, and not considering their relative size or shape for identification. “Photos which require understanding an object’s shape seem most likely to trick a model,” says Hendrycks.

With ImageNet-A, researchers have successfully found 7,000 blind spots in vision AI. Does that mean these images can just go into a new training set and fix their shortcomings? Probably not. “As there is much diversity and complexity in the real world, training on these images would likely not teach models how to robustly manage the full range of visual inputs,” says Hendrycks. “Collecting and labeling, say, 1 trillion images may resolve some model blind spots, but patching each previous blind spot will likely fail when novel scenarios occur and when the world changes.”

In other words, simply adding more and more photos to current machine learning data sets cannot fix the core shortcomings of its logic. There will always be images a computer hasn’t seen enough of before to identify accurately. So what can researchers do to close that lingering 5% gap? Hendrycks says they need to develop new methodologies, outside the bounds of modern-day machine learning, to create more sophisticated AI systems. Or, you know, they could not do that—and let us humans keep our smug superiority over machines for just a little bit longer.


  1. I’ll immediately seize your rss feed as I can’t find your e-mail subscription link or e-newsletter service.
    Do you’ve any? Kindly allow me recognise in order that I
    may subscribe. Thanks.

  2. My family members all the time say that I am
    wasting my time here at net, but I know I am getting experience all the time by reading thes good articles.

  3. Thank you for any other wonderful post. The place else could anyone get that type of information in such a perfect approach of writing?
    I have a presentation next week, and I am at the look for
    such info.

  4. I visited several web sites but the audio feature for audio songs present at this web site is actually marvelous.

  5. Greetings! I know this is kind of off topic but I was
    wondering if you knew where I could locate a captcha plugin for my comment form?
    I’m using the same blog platform as yours and I’m having difficulty
    finding one? Thanks a lot!

  6. Ahaa, its nice discussion regarding this paragraph at this place at this
    web site, I have read all that, so now me also commenting here.

  7. I was suggested this blog through my cousin. I’m now not
    positive whether this publish is written via him as nobody
    else understand such special about my trouble. You’re amazing!

    Thank you!

  8. We absolutely love your blog and find most of your post’s to be just what I’m looking for.
    can you offer guest writers to write content for you personally?
    I wouldn’t mind writing a post or elaborating on a
    lot of the subjects you write with regards to here. Again, awesome weblog!

  9. My spouse and I absolutely love your blog and find nearly all of your post’s to be just
    what I’m looking for. Does one offer guest writers to write content
    for you personally? I wouldn’t mind writing a post or elaborating on a lot of the subjects
    you write with regards to here. Again, awesome site!

  10. Good site you have here.. It’s hard to find high quality writing like yours
    these days. I honestly appreciate individuals like you!
    Take care!!

  11. Hi there this is somewhat of off topic but I was wondering if blogs use WYSIWYG editors or if you have to manually code with HTML.
    I’m starting a blog soon but have no coding skills so I wanted to
    get guidance from someone with experience. Any help would be enormously appreciated!

  12. Howdy! This post couldn’t be written any better!
    Reading this post reminds me of my previous room mate!
    He always kept talking about this. I will forward this page to him.

    Pretty sure he will have a good read. Thank you for sharing!

  13. I have been exploring for a little bit for any high-quality articles or weblog posts on this kind of area .
    Exploring in Yahoo I eventually stumbled upon this web site.
    Studying this information So i’m satisfied to convey that I have a very excellent uncanny feeling I found out just what
    I needed. I most indubitably will make certain to don?t disregard
    this site and give it a glance regularly.

  14. I all the time used to study paragraph in news papers but now as I am a
    user of internet so from now I am using net for articles, thanks to web.

  15. I have fun with, lead to I found just what I was taking a look for.
    You have ended my 4 day long hunt! God Bless you man. Have a nice day.

  16. Hello, i think that i saw you visited my website so i came
    to “return the favor”.I’m attempting to find things to enhance my site!I suppose its ok to use a few of your ideas!!

  17. Outstanding post however , I was wanting to know if you could write a litte more on this subject?

    I’d be very grateful if you could elaborate a little bit further.
    Appreciate it!

  18. Hey there would you mind letting me know which hosting company
    you’re working with? I’ve loaded your blog
    in 3 completely different internet browsers and I must say this
    blog loads a lot faster then most. Can you suggest a good hosting provider at a fair price?
    Kudos, I appreciate it!

  19. I do accept as true with all of the ideas you’ve presented
    for your post. They’re very convincing and will certainly
    work. Still, the posts are too brief for starters. May you please extend them a little
    from subsequent time? Thanks for the post.

  20. Hello to all, since I am really eager of reading this web
    site’s post to be updated regularly. It carries fastidious information.

  21. Pretty section of content. I simply stumbled upon your weblog and in accession capital to claim that I get actually loved
    account your blog posts. Any way I’ll be subscribing to your feeds and even I achievement you get
    right of entry to persistently fast.

  22. I’m impressed, I have to admit. Seldom do I come
    across a blog that’s both equally educative and engaging, and let me tell you, you have hit the nail on the head.

    The issue is something not enough folks are speaking
    intelligently about. I’m very happy I came across this in my hunt for
    something regarding this.

  23. Greetings! I know this is kinda off topic but I was wondering if you knew where I could find a captcha plugin for
    my comment form? I’m using the same blog platform as yours and I’m having trouble finding
    one? Thanks a lot!

  24. I’ll right away take hold of your rss feed as I can’t to find your e-mail subscription hyperlink or e-newsletter
    service. Do you have any? Kindly allow me know so that I
    may just subscribe. Thanks.

  25. An intriguing discussion is worth comment. I do think that
    you ought to publish more on this subject, it might not be a
    taboo matter but generally people don’t speak about such issues.

    To the next! Kind regards!!

  26. I think this is one of the most vital information for me.
    And i am glad reading your article. But wanna remark
    on few general things, The web site style is wonderful, the articles is
    really excellent : D. Good job, cheers

  27. I blog quite often and I truly thank you for your
    content. The article has truly peaked my interest.
    I’m going to bookmark your blog and keep checking for new information about
    once a week. I subscribed to your Feed as well.

  28. Nice blog! Is your theme custom made or did you download it from somewhere?
    A design like yours with a few simple tweeks
    would really make my blog stand out. Please let me know where you got your design. Appreciate it

  29. Thank you for every other magnificent post. The place else could anyone get that type of information in such a
    perfect method of writing? I have a presentation subsequent week, and I’m at the look for such information.

  30. First of all I would like to say terrific blog!
    I had a quick question in which I’d like to ask if
    you don’t mind. I was interested to know how you center yourself and clear your thoughts prior to writing.
    I have had a hard time clearing my mind in getting my thoughts out there.
    I do enjoy writing however it just seems like the first 10 to 15 minutes are
    usually lost just trying to figure out how to
    begin. Any recommendations or tips? Appreciate it!

  31. Hi there, this weekend is good in support of me, as this occasion i am reading this enormous educational paragraph here at my

  32. Hurrah! In the end I got a blog from where I be capable of really obtain valuable facts regarding my study and

  33. Hi everyone, it’s my first pay a quick visit at this website, and post
    is actually fruitful in support of me, keep up posting such posts.

  34. Nice post. I learn something totally new and challenging on blogs I
    stumbleupon everyday. It’s always helpful to read through articles from other
    writers and practice something from their sites.

  35. Do you have a spam issue on this blog; I also am
    a blogger, and I was wondering your situation; many of us have created some
    nice procedures and we are looking to swap solutions
    with other folks, why not shoot me an email if interested.

  36. For most recent news you have to go to see internet and on the web I found this site as a most excellent site for most recent updates.

  37. I love reading through an article that can make men and women think.
    Also, thanks for allowing for me to comment!

  38. I was suggested this web site by my cousin. I’m not sure whether this post is written by him as nobody else know such detailed about
    my difficulty. You’re incredible! Thanks!

  39. I’ve been browsing online greater than three hours nowadays, but I never discovered any attention-grabbing
    article like yours. It is pretty price enough for me. In my opinion,
    if all web owners and bloggers made just right content as you did, the net can be
    much more useful than ever before.

  40. Great website you have here but I was wanting to know if you knew of any message boards that cover the same topics discussed here?
    I’d really love to be a part of community where I can get advice from other experienced individuals that share the
    same interest. If you have any suggestions, please let me know.

    Many thanks!

  41. Hey There. I discovered your blog the use of msn. This is a really neatly written article.
    I will be sure to bookmark it and return to read more of your helpful info.
    Thanks for the post. I will definitely return.

  42. This is the perfect site for everyone who really
    wants to find out about this topic. You know a whole lot its almost tough
    to argue with you (not that I actually will need to…HaHa).
    You certainly put a brand new spin on a topic that has been discussed for
    many years. Great stuff, just great!

  43. Undeniably consider that that you stated. Your favourite
    reason seemed to be at the net the easiest factor
    to be aware of. I say to you, I definitely get irked whilst folks think about concerns that they plainly don’t realize about.

    You managed to hit the nail upon the highest and also outlined out the entire thing with no need side effect , other folks can take a signal.

    Will probably be again to get more. Thank you

  44. May I just say what a comfort to discover someone who actually knows what they’re discussing on the internet.
    You definitely realize how to bring a problem to light and make it
    important. A lot more people need to read this and understand this side of your story.
    I can’t believe you aren’t more popular given that you
    definitely have the gift.

  45. Hello to all, as I am genuinely keen of reading this website’s post to be updated regularly.
    It carries nice stuff.

  46. I’m truly enjoying the design and layout of your website.
    It’s a very easy on the eyes which makes it much more enjoyable for me to come
    here and visit more often. Did you hire out a designer
    to create your theme? Outstanding work!

  47. Asking questions are in fact good thing if you aare not understanding something totally, except this piece of
    writing provides nice understanding even.

    Heere is my web-site; Verla

  48. Hello there! Quick question that’s entirely off topic. Do you know how to make your site mobile
    friendly? My weblog looks weird when browsing
    from my iphone. I’m trying to find a template or plugin that might be able to correct this issue.
    If you have any recommendations, please share. Thanks!

  49. I will immediately snatch your rss as I can not to find your email subscription link or newsletter service.

    Do you’ve any? Kindly permit me understand in order that I may just
    subscribe. Thanks.

  50. Howdy would you mind stating which blog platform you’re using?
    I’m looking to start my own blog soon but I’m having
    a tough time making a decision between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your layout seems different then most blogs and I’m looking for something unique.
    P.S Apologies for being off-topic but I had to ask!

  51. It’s really a great andd useful piece oof info. I am happy that you simply shared this useful info with us.
    Please keep us informed like this. Thanks for sharing.

    Here is my site … teen nude

Leave a Reply

Your email address will not be published.