The open source AgentLens benchmark pairs objective test checks with LLM written reviews of each coding agent run and ships as a nightly regression pipeline, not a static leaderboard.
Most AI coding agents are still graded like a math test: pass or fail. The developers who actually use them see something messier: a whole trajectory of tool calls, half-finished explanations, recovered mistakes, and silent assumptions that never show up in a score.
A new open-source benchmark called AgentLens, posted to arXiv in July, is built to capture that messiness. It evaluates interactive coding agents on the full sequence of decisions an agent makes on a task, not just whether the final patch passes the test suite.
Each run is scored twice. A formal-verification layer runs objective checks: unit tests, repo-state diffs, regex matches, builds, and static analysis, and produces a binary-style outcome. An LLM judge then writes a trajectory review, citing the specific tool calls, edits, or assumptions that drove the result. The two scores are presented together, with the review explaining why the objective outcome happened the way it did. A pass with a flaky test reads as different from a pass with a clean refactor. A fail that recovered after five retries reads as different from a fail that silently dropped a requirement. Pass/fail benchmarks collapse all four cases into the same bit. AgentLens keeps them apart by attaching an evidence-cited explanation to every score.
A trajectory, in the paper's framing, is the full log of an agent's run: every file read, every command issued, every edit made, and every error recovered from. That log is what the LLM judge reads when it writes its review, and it is also what the benchmark replays for side-by-side comparisons. Pairing a test-driven outcome with a written review grounded in the same evidence is the design move that distinguishes AgentLens from prior coding-agent evaluations like SWE-Bench, which report a binary patch-applied result with no read of how the agent got there.
The same trajectories also drive side-by-side comparisons against an anchor run, so a team can ask whether last night's model swap actually changed behavior, not just whether the new model happened to pass more tests. The repo, agent-lens/agent-lens-bench, ships with a Java fold leaderboard CSV and is positioned as a nightly regression pipeline rather than a static ranking. The intended use cases, per the paper, are diagnosing where a model version breaks down, comparing successive releases of the same agent, and catching product regressions before they ship.
The design has a production-shaped bias. Side-by-side comparisons are built around the same task set across versions, which is what a CI pipeline needs. The LLM review lane is meant to be read by a human engineer reviewing a failed nightly run, not by a leaderboard crawler racing models. That positioning tracks how production teams already think about these systems: less as contestants in a benchmark tournament, more as components in software that can regress between deploys. It is also why the paper treats the contribution as diagnostic and regression-catching rather than a replacement for existing benchmarks.
The second-order effect is the kind of failure mode the rubric is built to surface. A pass that hinged on a hard-coded test stub, a recovery loop that masked a real bug, or a refactor that quietly bypassed a guardrail all look identical on a binary score and very different in a trajectory review. For teams running agents against live codebases, that distinction is the difference between a green nightly that ships and a green nightly that broke the build in a way the test suite cannot see.
Two structural limits matter. The leaderboard CSV is a Java fold snapshot and does not yet cover other languages or task domains, so model rankings from it should not be generalized across the coding-agent space. The trajectory reviews are LLM-written, which means the rubric is subjective by construction, and the same authors also build a coding agent, so the review lane carries some self-evaluation risk. Both caveats are spelled out in the paper itself.
The release lands as coding agents are leaving chat demos and entering CI pipelines. The question teams now need answered is no longer whether the agent passed the test, but what it actually did, and whether the new version is trustworthy enough to keep running. Trajectory review is one answer to that question, not the final one. Next to watch: whether the open leaderboard expands beyond the Java fold, and whether independent teams adopt the trajectory-review format for their own nightly pipelines.