Microsoft Announces 'Computer Use' Automation in Copilot Studio
Microsoft has introduced a new AI-powered feature called "Computer Use" for its Copilot Studio platform that allows agents to directly interact with websites and desktop applications using simulated mouse clicks, menu selections, and text inputs.
According to Charles Lamanna, corporate vice president of business and industry for Microsoft Copilot, the feature marks a major leap forward in Microsoft's vision for intelligent automation by allowing Copilot Studio agents to treat graphical user interfaces similarly to how a human user would.
"Computer Use adapts to changes in apps and websites automatically," said Lamanna. "It adjusts in real time using built-in reasoning to fix issues on its own, so work continues without interruption. It is also built on Copilot Studio's robust security measures and governance frameworks, to help ensure compliance with organizational and industry standards."
Key use cases for the new feature include:
- Automated data entry across systems that lack direct integrations;
- Market research automation by collecting and organizing information from online sources; and
- Invoice processing, streamlining finance operations by extracting and transferring data into accounting systems.
Computer Use is hosted on Microsoft infrastructure. The company said this eliminates the need for organizations to provision or manage dedicated RPA environments, reducing both overhead and maintenance requirements.
Microsoft says the new feature also modernizes traditional robotic process automation (RPA) by making it more resilient and accessible. Rather than requiring extensive coding or scripting, users can describe automation tasks using natural language and refine workflows with side-by-side video previews showing how the agent interprets and executes UI actions.
Additional enhancements include full transparency, with the ability to audit every action the agent performs, including screenshots and reasoning steps.
Microsoft plans to showcase Computer Use in greater detail next month at its Build conference.
Microsoft Begins Rolling Out Controversial Recall Feature
In other Microsoft Copilot news, the company recently announced it has begun rolling out a preview of its Recall feature in Windows in the latest Windows 11 preview build.
Microsoft's Recall feature for Copilot+ PCs offers an AI-driven way to search and retrieve past activity across apps, websites, documents, and images by describing what you remember. The feature works by taking opt-in snapshots of your screen throughout the day and requires Windows Hello authentication to ensure only the device owner can access them.
The feature was initially supposed to start rolling out in preview last June. However, after security concerns from the community arose, the feature was yanked. It was then scheduled for release in October, but was yet again pulled when the feature was found to be taking screengrabs of users' credit card information.
Microsoft has made it clear that Recall is an optional feature and can be turned off at any time. It also said the new version now rolling out adheres to the company's commitment to AI and privacy.
"Recall does not share snapshots or associated data with Microsoft or third parties, nor is it shared between different Windows users on the same device. Windows will ask for your permission before saving snapshots. You are always in control, and you can delete snapshots, pause or turn them off at any time. Any future options for the user to share data will require fully informed explicit action by the user," said Microsoft.