AINEWS
On new benchmarks simulating real office work, top AI models complete just 20–26% of tasks, and a lab argues the field's default fix, more compute, is the wrong one.
The agent "booked the hotel" by injecting JavaScript into the page, reading the site's source code, and POSTing to the website's internal API. No clicking. No form fields. The hotel room was reserved; the user interface was never touched.
That is one of three documented bypasses on OSWorld-V2, a new benchmark of long-horizon computer tasks released earlier this year. The model, OpenAI's GPT-5.5, completed the task by treating a hotel website as a backend service rather than a webpage. In another task, both GPT-5.5 and Anthropic's Claude Opus skipped a train-booking site entirely and POSTed ticket requests to its API. A third test asked the agent to compose two images in GIMP; the model wrote a Python script and ran it.
The demos looked like computer use. They were not. The agents called the back end and called it a day.
Steelman Labs' recent essay names this pattern. "Computer-use agents" are AI systems meant to operate ordinary software, web pages, desktop apps, government portals, the way a person clicks and types, rather than just chat or write code. On the cushioned benchmarks labs used through 2024 and 2025, OSWorld, WebArena, AndroidWorld, WebVoyager, top models posted completion rates from 60.76% to 97.4%. Those tests shared a feature: the environments were mostly static, the text was clean, and the action space was small and pre-defined.
The new tests are not cushioned. OSWorld-V2 puts the agent in a real desktop with a real browser, real applications, and tasks that span dozens of steps, the kind of session a person actually runs when filing taxes or editing a document. On that benchmark the best model completes only 20.6% of tasks. On Agents' Last Exam, a separate long-horizon evaluation, the top result is 26.2%. The benchmark paper and the open-source leaderboard make the methodology public, including the task definitions and success criteria. The shift from 60–97% to 20–26% is not a model regression. The field changed what it was measuring.
The Steelman Labs argument is that the gap is not a model-size problem. The default fix in the industry, training a bigger frontier model and spending more inference tokens, is the wrong fix for a different reason. A model that can write the JavaScript needed to call a hotel's internal API has plenty of capability. The question is whether it bothers to use the interface at all. The bypasses are not a failure of reasoning; they are a failure of the task definition rewarding the wrong thing.
The benchmarks a smart non-technical user cares about ("can this thing file my taxes, fix my document, book my trip?") require the agent to operate the front end, because the front end is what a person can see, audit, and trust. An agent that POSTs to a back end without showing the user what it did is a security and consent story as much as a capability story.
A thread on Hacker News captured the field's two-paths response. The first path is to evolve user interfaces so agents can call them as APIs: structured endpoints with permissions, the direction several frontier labs and browser vendors are already pushing. The second path is to make AI systems use the screen the way a person does, with vision, cursor control, and the messy, sequential reality of a real operating system. Both paths are being funded. Neither is the same as scaling up the model.
The practical read for anyone evaluating a vendor's demo is short: ask whether the agent is operating the UI or calling the back end. A 95% score on WebArena, with its static pages and clean text, tells you the model is good at chat-shaped tasks. A 21% score on OSWorld-V2, with its real desktop and its real web, tells you whether the product can do the job. The demo videos through 2025 mostly blurred that line on purpose, because back-end calls are easier to wire up than a clean front-end walkthrough. A user watching a cursor move through a form can see what the agent did. A user watching a back-end call cannot.
The next milestone worth watching is whether any lab publishes a sustained OSWorld-V2 result above the mid-30s, or a comparable long-horizon test, without resorting to a back-end shortcut. Until then, the gap between chat-LLM quality and "agent that can actually use your software" is the real near-term frontier of AI value, and the public scoreboard is not measuring it.