Gemini Can Now Answer Questions About What’s on Your Screen

UPDATE (3/24): After announcing two screen-reading capabilities for Gemini at MWC last month, Google is now rolling them out to some Android users.

On Sunday, a Reddit user shared a demo of Gemini’s “Share screen with Live” feature, which allows people to ask questions about what’s displayed on their screen. The other Gemini feature, which launches the camera app and lets users ask questions about what they are viewing, is also being released, Google confirmed to The Verge. Both features are part of Google’s Project Astra and require a Gemini Advanced subscription.

Original Story (3/3):
Google has announced new screen-sharing and live video capabilities for Gemini Live, its competitor to ChatGPT’s voice mode. These features allow users to share their screens with Gemini and ask questions based on what’s displayed.

In a video presentation at Mobile World Congress, Google demonstrated how the new “Share screen with Live” button in the Gemini mobile app works. A user taps the button, summons Gemini, and asks what would pair well with a pair of jeans displayed on their screen. This is followed by a back-and-forth conversation, ending with Gemini providing the user with some clothing suggestions. 

In a similar demo for the live video sharing feature, tapping the Live button (voice mode) on the Gemini app’s home page led to an interface with a camera button. Enabling the camera from here allowed the user to share their view with Gemini and ask questions about it—as if they were on a video call with Gemini.

Both features are part of Project Astra, Google’s multimodal AI assistant that can detect what’s around it and converse with users. When Google teased its viewing ability at the I/O event in May, it could identify what it was looking at and provide intelligent responses through voice interactions with the user.

Google had planned to release parts of the project by the end of last year, but it seems to have been delayed by a few months. The screen-sharing and live video features of Gemini Live will now roll out to Gemini Advanced subscribers on Android later this month. Those at the MWC in Barcelona can check out the features at Android Avenue between Halls 2 and 3 right now. 

In addition to Project Astra, Google is rumored to be working on another Gemini-based agentic feature for Chrome called Project Jarvis. According to The Information, Jarvis can memorize your regular web-based tasks and complete actions autonomously.

About Jibin Joseph

Contributor

Jibin is a tech news writer based out of Ahmedabad, India. Previously, he served as the editor of iGeeksBlog and is a self-proclaimed tech enthusiast who loves breaking down complex information for a broader audience.

Read Jibin’s full bio

Read the latest from Jibin Joseph