Google has taught AI Mode to search by images, but not for all users yet

Last month, Google introduced AI Mode, a search chatbot powered by artificial intelligence that is integrated into the company’s proprietary app. Now, AI Mode has learned to “see” images and answer questions about them. This innovation is already available to “millions of new users.”

Image source: BoliviaInteligente / Unsplash

The search chatbot update combines a custom version of the large Gemini language model with Lens image recognition technology. This lets users take a screenshot of something or upload an image to get a “rich, comprehensive, and linked answer” about what’s in the original file. The feature is available in the Google app for Android and iOS devices starting today.

A Google spokesperson noted that AI Mode builds on the company’s years of work on visual search, which allowed it to take a step forward. He also added that Gemini’s multimodal capabilities allow the chatbot to understand the entire scene in an image, including the context of how objects relate to each other, their shapes, colors, locations, and more.

According to Google, the updated algorithm uses a “fan technique,” ​​in which the neural network sends multiple queries to an image and the objects in it. The result is “incredibly nuanced and contextually relevant” responses. Last month, Google launched the AI ​​Mode bot exclusively for Google One AI Premium subscribers. Now, the feature is available to more users in the U.S.