Anthropic Releases Claude 3.5 Sonnet and Haiku, Expands AI with 'Computer Use' Public Beta

• By John K. Waters
• 11/07/24

Anthropic has unveiled two advanced AI models, Claude 3.5 Sonnet and Claude 3.5 Haiku, with significant improvements in functionality and performance, especially in coding. The upgraded Claude 3.5 Sonnet outperforms its predecessor in several benchmarks, the company said, while Claude 3.5 Haiku matches the intelligence of Anthropic's previous largest model, Claude 3 Opus, at a similar cost and speed.

Claude 3.5 Sonnet brings notable enhancements in coding. It improves its SWE-bench Verified score to 49.0%, outperforming other publicly available models and specialized systems. The model also advances in TAU-bench, an agentic tool use task, showing gains in both retail and airline domains.

GitLab, a web-based platform that helps teams collaborate on software development, tested the model for DevSecOps tasks and found it delivered stronger reasoning (up to 10% across use cases) with no added latency, making it an ideal choice to power multi-step software development processes.

The release announcement included support statements from key clients, including Cognition and The Browser Company, which highlighted the model's improvements in coding, planning, and automation. "Claude 3.5 Sonnet has outperformed every model we've tested in automating web-based workflows," a Browser Company spokesperson said.

Claude 3.5 Haiku, Anthropic's next-generation fast model, offers broad performance improvements at the same cost and speed as the previous Claude 3 Haiku. Scoring 40.6% on SWE-bench Verified, it surpasses older models, making it suitable for user-facing products and tasks requiring personalization, such as analyzing purchase history or managing inventory records. Claude 3.5 Haiku will launch later this month on Anthropic's API, Amazon Bedrock, and Google Cloud's Vertex AI.

Anthropic also introduced a new "computer use" capability in public beta. Claude 3.5 Sonnet is the first model to offer this feature, which enables the AI to simulate human interactions with computer interfaces — moving cursors, clicking, and typing. Initially experimental, this function aims to automate complex, multi-step tasks for developers. Replit, for example, is using Claude's UI navigation abilities for in-progress evaluations in their Replit Agent product.

Claude's OSWorld evaluation score — 14.9% in the screenshot-only category and 22.0% with additional steps — highlights its potential to mimic human-like computer operations. (OSWorld provides benchmarking multimodal agents for open-ended tasks in real computer environments.) However, Anthropic advises developers to start with low-risk applications, as the technology may occasionally struggle with tasks like scrolling and zooming. "This capability is imperfect but rapidly evolving, and we are proactive in its safe deployment," Anthropic stated, underscoring new classifiers developed to detect misuse and mitigate risks.

In collaboration with the US and UK AI Safety Institutes, Anthropic conducted pre-deployment testing to ensure Claude 3.5 Sonnet adheres to ASL-2 safety standards under its Responsible Scaling Policy. The company emphasizes a commitment to safe AI evolution, recognizing both the potential and implications of more capable systems.

"We're eager for developers to explore these advancements and provide feedback. This is just the beginning of a new chapter in working with Claude," Anthropic said.