For smartphones and glasses: Google Gemini to become a universal AI assistant
Google is experimenting with new Gemini functions in Project Astra. In future, the assistant will even be able to ask the bike dealer for spare parts.
Google Gemini should be able to make calls for us at some point.
(Image: Google)
Google's AI team at Deepmind is working on 'Project Astra', a universal AI that can take over tasks from users in the long-term thanks to agents. This sounds familiar at first, as Google has been promising this for years – first for Google Now and then for the Assistant. However, the tasks and functions that should one day be possible range from assistance with bicycle repairs and shopping to tutoring pupils. At Google I/O, the company showed a foretaste of what will soon be possible on smartphones and, in the future, glasses.
With Project Astra, Google had already presented a “research prototype for a universal AI assistant” at I/O 2024, some of whose announced functions are already part of the Gemini AI assistant. For example, Gemini can access the camera and screen content via the live function.
According to Google, Gemini Live can help with a blocked drain, for example, or give tips on how to fix a broken record player yourself. To get this advice from Gemini, the user simply points the smartphone camera at the object and asks the AI what the problem might be. Gemini can also remember things that you have previously told the AI. Google, ChatGPT, and other AI providers consider this memory function to be important because an assistant can only be helpful if it knows the user, according to OpenAI and others.
Gemini: Google shows where the AI journey is headed with Project Astra
During this year's I/O, the company demonstrated further potential functions of Astra that could eventually become part of Gemini. In addition to a better understanding of natural language, an significant ingredient will be agents that proactively do things for the user. In this context, proactive means that the assistant draws the user's attention to things, as they may not yet know what they will need in the next step. This is another reason why memory is so important.
A video in which a man wants to repair a bicycle with the help of Astra on his smartphone shows how the company envisions this. He asks the AI to look up an instruction manual for his bike and open the chapter on brakes. The assistant then independently scrolls through the PDF to the desired point. However, the research version of the assistant can do much more.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
For example, in response to a verbal request, the AI searches for a YouTube video that explains how to unscrew a frayed screw. Gemini will also be able to call a bike store for the user to ask for a spare part – Google uses AI agents to delegate tasks. With the help of the smartphone camera, the assistant will also assist in the search for a suitable screw nut.
(Image:Â Google)
The assistant has obtained the relevant information from the email sent to the user by the bike dealer and now uses an AR overlay on the smartphone display to show which box contains the correct nut. The prerequisite here is that the workshop is tidy and everything is labeled. As with a normal conversation partner, the conversation can be interrupted and then resumed.
Another video shows how Gemini could help with learning in the future. A girl takes a screenshot of a chemistry problem she doesn't understand and asks Gemini for help. The AI goes through the topic with her and explains the solution step by step. In another task, the user asks what she did wrong in her task – the AI points out the mistake and circles it on the smartphone screen.
Empfohlener redaktioneller Inhalt
Mit Ihrer Zustimmung wird hier ein externes YouTube-Video (Google Ireland Limited) geladen.
Ich bin damit einverstanden, dass mir externe Inhalte angezeigt werden. Damit können personenbezogene Daten an Drittplattformen (Google Ireland Limited) übermittelt werden. Mehr dazu in unserer Datenschutzerklärung.
Gemini: Proactive assistant for smartphones and glasses
Google uses the clips to demonstrate how the company envisions the type of interaction with the AI. Gemini should ultimately be a real assistant for every situation and be able to carry out research tasks and even make calls for the user. Google had actually already demonstrated the latter function with Duplex during I/O in 2018, but the company later admitted that it had cheated during the demo. Duplex could still be used for Gemini's agent functions. However, Google is no longer using the name. However, Duplex is used in the “Aks for me” function introduced in the USA at the beginning of 2025, which allows users to automate appointments.
Videos by heise
The scenarios demonstrated could make everyday life easier. Of course. However, it remains to be seen when Google will integrate these functions into Gemini and how well and to what extent this will actually work. As indicated in the bike repair demo, the AI should have access to a lot of our data in order to be really helpful. However, open access to data by users always carries risks and requires the user to have great trust in the company. Although Google is working on processing data locally on the smartphone, the current range of functions for on-device AI tasks is still quite limited.
(afl)
