Copilot Vision, a feature that allows Microsoft’s proprietary AI assistant to view and understand what’s happening on a user’s device screen, is now available for free in the Edge browser, according to Microsoft AI chief Mustafa Suleyman.
Image source: Microsoft
He also said that the Copilot Vision function allows the AI algorithm to literally see what the user sees on the screen. This innovation can be useful in various situations, for example, Copilot can suggest a recipe for cooking a dish displayed on the screen. However, it will not independently follow links or perform any actions on behalf of the user.
More advanced system-wide features of Copilot Vision will remain available only to paid subscribers of Copilot Pro. With this subscription, Copilot Vision goes beyond Edge and can help when interacting with other applications. For example, the AI assistant will tell you how to use a particular Photoshop function, tell you how to get through a difficult part of a game, and more.
To start using Copilot Vision in your browser, you need to open the corresponding section on the Microsoft website using Edge and confirm your consent to use the AI function. After that, it will be enough to open the Copilot sidebar and click on the shortcut with the image of a microphone. The start of the Copilot Vision function will be accompanied by a characteristic sound signal, and the browser will also change color scheme slightly. According to available data, Microsoft records Copilot responses, but does not collect user information, images or page content during interaction with Copilot Vision. To disable the function, simply close the browser window.