When AI grades AI, a false pass ships a broken agent

The most expensive mistake an AI judge can make is not getting a number wrong. It is waving a broken agent through. On a public benchmark for AI agents, a method that builds its own grading criteria cut that false-pass rate from 17.3% to 11.5%, roughly a third fewer broken agents waved through. The catch: its overall accuracy edge over a standard AI judge is not statistically significant.

The result, posted this week as an arXiv preprint, is a small win on the metric the field mostly ignores and a flat outcome on the metric the field mostly reports. That trade-off is the story.

The setup uses tau-bench, a public benchmark where an AI agent has to complete realistic customer-service tasks like booking a flight or fixing an account, and WebShop, a benchmark where the agent picks products from a mock storefront. The grader has to decide, after the agent finishes, whether the task actually got done. Researchers label 173 tau-bench trajectories by hand, then run a method called RubricForge that reads those labeled trajectories and writes its own checklist, in plain English, for what counts as success. The grader then applies that frozen checklist to held-out trajectories. The whole pipeline uses one frozen 7-billion-parameter model as both the agent and the judge, and it never touches the environment again once the rubric is written.

Other LLM-as-judge methods typically use either a hand-written checklist, the generic G-Eval style rubric, or a model fine-tuned on labeled examples. Both of those routes tend to credit fluent but unsuccessful trajectories, like an agent that politely confirms a booking it never made. A human-readable rubric, by contrast, names the criteria, so every verdict is traceable to a sentence in a checklist. That is structurally different from a fine-tuned judge whose reasoning lives in millions of opaque weights.

The rubric is built by reflective evolution. The system starts with a draft checklist, asks the model to grade labeled trajectories against it, compares those grades to the ground truth, and rewrites the rubric to reduce the disagreement. The process is "reward-free" in the sense that the only signal is agreement with the labeled examples, not a downstream score from a running environment. Once the rubric stops improving, it is frozen and used as-is. That is one model call per trajectory at evaluation time.

On tau-bench, RubricForge's false-pass rate is 0.115 versus 0.173 for the generic G-Eval baseline. False-pass rate is the metric that decides whether a broken product ships: a false pass lets an agent out the door that should have been caught, while a false fail only costs a retry. On WebShop, where the metric is rank correlation (Spearman, a measure of how well the judge's ordering of trajectories matches the ground-truth ordering), RubricForge's score rises from 0.370 to 0.410.

Two caveats matter. First, the raw agreement rate with ground truth on tau-bench is not statistically distinguishable from the generic judge: a McNemar test, which checks whether two judges make the same mistakes at the same rate, returns p = 0.248, well above the 0.05 threshold. Second, absolute-score calibration, meaning how close the numerical scores are to the right values, marginally favors the generic judge, with a mean absolute error difference of −0.048 (p = 2×10^−4). The preprint is explicit about both: its central claim is that for a reward-free evaluator, the false-pass rate, not aggregate agreement, is the deployment-relevant quantity.

Switching the yardstick from aggregate agreement to false-pass rate is the paper's contribution. LLM-as-judge systems are usually evaluated on agreement with human labels or with a reference judge. That treats all mistakes as symmetric, which they are not. A judge that is too generous ships broken products; a judge that is too strict wastes engineer hours. Optimizing for the asymmetric cost is a different optimization problem, and the result here is a worked example of what changes when you do.

The method runs on a single 7B model and two benchmarks. The remaining question is whether the rubric-evolution loop generalizes to larger judges and to agent settings where the failure modes are less obvious than a missing booking confirmation. The authors do not claim a general-purpose upgrade. They claim a method that improves the failure mode that decides whether a broken agent reaches a user, and the data on tau-bench supports that narrower claim.