A New Hard Test for AI Agents: The Best Model Solves 28%, None Clear 30%

The strongest agentic AI configuration tested in a new study (GPT-5.5 paired with its Codex harness) solves 28.0% of an 82-task subset deliberately curated to be hard. None of the eight model-harness pairs examined clears 30%.

The subset, Harbor-Index, is one of three releases from Lin Shi and co-authors; the others are adapters that port agentic benchmarks behind a single interface, and a controlled cross-benchmark study of eight frontier models. The project is open-sourced under the harbor-framework GitHub organization and described in an arXiv preprint, and is built on four design principles: compact, diverse, challenging, and high quality.

"Agentic" here means software that takes multi-step actions in an environment (a browser, a shell, a code repository) and is judged on whether the task is completed, not on whether the right string is produced. Agent benchmarks have been harder to compare than chatbot benchmarks because a model's score can move several points depending on the harness that surrounds it, the way the model is given tool access, and how success is verified.

Harbor Adapters ports more than 80 existing agentic benchmarks behind a single interface, so any compatible agent can be evaluated against any of them without bespoke integration work. The authors validated the adapters through code review and parity experiments against the upstream benchmarks' original behavior.

On top of that infrastructure, the team ran a large-scale evaluation: eight models spanning capability tiers, run across 54 benchmarks, each model tested with a standard harness (Terminus-2) and with one of three native agent harnesses. The full study consumed roughly 226 billion tokens and more than $300,000 in compute.

Harbor-Index is a slice of that larger suite: 82 difficult tasks drawn from 29 benchmarks, filtered for difficulty, audited with both AI and human reviewers, and run through an audit-and-fix loop that distinguishes a model failure from a broken task (verifier mismatches, underspecified success conditions, brittle environments).

The authors also flag a cost caveat. Open-weight models routed through Claude Code over OpenRouter hit a low cache rate, which inflates their cost well above what the same models cost on Terminus-2. On the pass-rate-versus-cost chart, those configurations fall off the frontier, even when their pass rates are competitive.

The 28.0% figure is a defended ceiling, not a leaderboard ranking. It comes from the authors' own evaluation, the broader study has not been independently replicated, and the preprint notes that some final figures may shift in the camera-ready version. The release site lists 28.1% for the same configuration, a tenth-of-a-point gap.

The ceiling holds up because the audit-and-fix step checked every task. Most agent benchmarks have shipped with at least some tasks where the verifier rejects correct answers or the environment breaks under a plausible action. A benchmark whose top score is 30% is easy to dismiss as unfair; one whose top score is 30% after every task has been reviewed for verifier mismatches, underspecified success conditions, and brittle environments is harder to wave off.

Harbor-Index is positioned as a hard test, not a general-capability score. The authors contrast it with aggregate indices like the Epoch Capabilities Index, which collapse broad capability into a single number, and with score-prediction methods like BenchPress, which estimate performance without running the benchmark. Both approaches are useful; neither replaces running an agent against audited tasks.

The next round of agent releases is likely to cite Harbor-Index by name. When it does, the number to weigh is the gap between a model's reported score and 30%, and whether the score is reported on the curated subset or on the wider 54-benchmark suite. The new yardstick is public, the cost of running it is in the low hundreds of dollars per frontier agent, and the floor it draws is, in the authors' framing, the most defensible one available.