Microsoft upgraded Copilot: human voice, surfing with the user and deep thoughts

Microsoft has unveiled a radical update to its AI assistant, Copilot, adding vision and voice to make it more personalized. New Copilot features include a virtual news anchor mode that reads out headlines; voice function allows you to communicate with him naturally, like Advanced Voice Mode from OpenAI; finally, the AI ​​assistant can “see” what the user points it to.

Image source: blogs.microsoft.com

Microsoft Copilot has been radically redesigned on mobile devices, in the web version and in the Windows app – it now resembles the Pi AI assistant created by Inflection, a company that has a significant number of employees moving to Microsoft. Former Inflection CEO and Google DeepMind co-founder Mustafa Suleyman is now head of Microsoft’s AI department. The AI ​​assistant interface has been significantly redesigned – now it is a personalized Copilot Discover page, more useful and attractive than just a text field for queries to a chatbot. The home page is customized based on the user’s history with the chatbot; Over time, it will provide useful searches, tips, and relevant information.

The AI ​​assistant received a full-fledged voice interface similar to ChatGPT – now you can communicate with it, ask questions, interrupt it, as when talking with a friend or colleague. There are four voice options available in Copilot, one of which you will have to choose the first time you use it. Microsoft’s second big bet is Copilot Vision, which allows the assistant to see what the user sees on an open web page. You can ask questions about text, images, or any other content, and the assistant will give you a natural answer. This will help when shopping online – the system will offer various product options and reviews. You will have to connect Copilot Vision manually, and the content viewed by the system is not stored anywhere and is not used for AI training, Microsoft assures. Additionally, the company has placed restrictions on the types of sites Copilot Vision can work with; The system also does not support paid and confidential materials. In one of the examples given by Microsoft, the function is used to scan handwritten culinary recipes – the AI ​​explains what kind of dish it is about and how long it will take to prepare it.

Copilot Daily, another feature, presents an audio summary of news and weather that an AI assistant reads out like a newscaster. The summary is a short note that the user can listen to in the morning – only resources that have agreed to use the content by Copilot are used as sources. At the initial stage, these are Reuters, Axel Springer, Hearst and the Financial Times, but in the future their number will expand. The Think Deeper feature is also promising – thanks to the latest OpenAI AI models, Copilot can now answer complex questions, offering step-by-step answers or making comparisons. The feature remains at an early stage of development, and it is available on the Copilot Labs site along with Copilot Vision – test participants can send feedback to Microsoft on their work. The company is being cautious for good reason, given the concerns that the Recall feature caused before it was redesigned.

The updated Copilot package is available in mobile applications for iOS and Android, on the website copilot.microsoft.com, as well as in the Copilot application for Windows. Copilot Voice is initially available in English in Australia, Canada, New Zealand, the UK and the US, with future availability in other regions. Copilot Daily is currently only available in the US and UK, and Copilot Vision is limited to select Copilot Pro subscribers in the US.