Skip to main content

Google Lens now lets you search with video

Google Lens now lets you search with video

/

You can use your voice to ask questions about what you’re seeing, too.

Share this story

Image of the Google “G” logo on a blue, black, and purple background.
Illustration: The Verge

If you can’t capture what you want to search for with just a picture, Google Lens will now let you take a video — and even use your voice to ask about what you’re seeing. The feature will surface an AI Overview and search results based on the video’s contents and your question. It’s rolling out in Search Labs on Android and iOS today.

Google first previewed using video to search at I/O in May. As an example, Google says someone curious about the fish they’re seeing at an aquarium can hold up their phone to the exhibit, open the Google Lens app, and then hold down the shutter button. Once Lens starts recording, they can say their question: “Why are they swimming together?” Google Lens then uses the Gemini AI model to provide a response, similar to what you see in the GIF below.

Google will take your video and question into account when providing a response.
Google will take your video and question into account when providing a response.
GIF: Google

When speaking about the tech behind the feature, Rajan Patel, the vice president of engineering at Google, told The Verge that Google is capturing the video “as a series of image frames and then applying the same computer vision techniques” previously used in Lens. But Google is taking things a step further by passing the information to a “custom” Gemini model developed to “understand multiple frames in sequence... and then provide a response that is rooted in the web.”

There isn’t support for identifying the sounds in a video just yet — like if you’re trying to identify a bird you’re hearing — but Patel says that’s something Google has been “experimenting with.”

You can use your voice to ask a question about a photo, too.
You can use your voice to ask a question about a photo, too.
GIF: Google

Google Lens is also updating its photo search feature with the ability to ask a question using your voice. To try it, aim your camera at your subject, hold down the shutter button, and then ask your question. Before this change, you could only type your question into Lens after snapping a picture. Voice questions are rolling out globally on Android and iOS, but it’s only available in English for now.