Microsoft's Copilot now sees what you see in the Edge browser

Microsoft's head of AI Mustafa Suleyman enthuses at Bluesky about the fact that Copilot Vision is now available for Edge. This is “pretty exciting”. The function allows the browser to see what the user sees – but this only applies to the screen and, in reality, only to a few selected websites.

To use Copilot Vision, you have to agree to it or activate it; it is switched off by default. An initial similar function from Microsoft called Recall was met with massive protests because the idea of an AI that was constantly watching was not appreciated by humans.

Suleyman writes that with Copilot Vision, you can now talk to the AI about what you see. It no longer requires extensive explanations or copy-pasting. The function can be found in the browser behind the Copilot microphone icon as a glasses icon. However, it does not work everywhere. Microsoft names the selected partners: these are initially Wikipedia, Amazon, OpenTable, TripAdvisor and a few others that do not exist in the EU.

AI also gets eyes at Google, OpenAI, and Meta

In fact, other AI providers already have similar functions up their sleeves that work much more extensively. Google's Gemini Live, for example, can not only see what is on the screen of a smartphone, but also what you can see through the camera. However, this function is not available for all devices.

OpenAI offers Advanced Voice Mode, which basically has the same range of functions. Meta can use Meta AI to see what the wearer sees and react to it, thanks to Ray-Ban glasses, for example.

Microsoft also already offers a Copilot Vision function for Windows Insiders. Here, the AI can analyze content from open files and the browser. Here, too, you click on the glasses icon. The visual search is initially only available to people in the USA. An advanced search, which is also based on AI, is currently being made available to Windows users worldwide.

(emw)