Microsoft Starts Testing Copilot Vision — AI Will “See” the Screen and Applications on Your Computer

Microsoft has begun testing another update to its Copilot app for Windows that will allow it to broadcast your computer screen and the interface of apps running on it to an AI assistant. Initially, the Copilot Vision feature was limited to the Edge browser, but it is now extended to any app on your PC.

Image source: blogs.windows.com

Copilot Vision can help you understand how Adobe Photoshop works, analyze photos or web pages that the user opens. At the event dedicated to Microsoft’s 50th anniversary, the company showed how the assistant helps you play Minecraft and optimize settings in the video editor Clipchamp. Only participants in the Windows Insider program from the United States can test the preliminary version of Copilot Vision.

Copilot can highlight specific areas of the screen to help users navigate a program’s interface. In practice, this is partly similar to the Recall feature, which automatically takes screenshots and allows you to search through them, and partly similar to screen sharing via Microsoft Teams. Copilot for Windows has also started testing AI file search, which allows you to ask the assistant questions about their contents; the supported formats are .DOCX, .XLSX, .PPTX, .TXT, .PDF, and .JSON. Both of these features are available for all Copilot for Windows PCs, not just Copilot Plus PCs.

Copilot Vision is also available in iOS and Android apps. On PC, the feature is currently being tested in the Windows Insider program, but may become generally available in the coming weeks or months.