AI agents can get more reliable after each run, without retraining

A research team proposes a different path to AI agent reliability: instead of retraining the model, teach the scaffolding around the agent to keep a running repair log. On eight interactive benchmarks, a procedural-memory layer they call Living-Harness added about 10 percentage points to first-try task success with the underlying model held fixed, and a companion paper this month contests the evaluation methodology those numbers rest on.

The paper, posted to arXiv as "Living-Harness Is an Interactive-Agent Evolver", frames the layer as a self-evolving harness, the bundle of tools, prompts, memory, and workflow rules that sits on top of a model when it acts as an agent. The harness is the part developers can edit after a model is deployed. The team's move is to make the harness learn from each completed interaction rather than stay static.

The mechanism is governed by what the authors call an Evolution-SOP, a written operating procedure the layer updates after each run. When an agent finishes a task, the system extracts an episode abstraction and structured update evidence, then writes two forms of procedural knowledge: episodic memory, with trigger conditions, failure patterns, and recovery actions; and a state graph, with nodes for the agent's situation, edges for repairs, and rules for transitions. The model, the tools, and the base context stay frozen. Only the procedural repairs accumulate across cycles.

When the next task arrives, the layer retrieves the updated harness state and uses it to guide the run. The authors also support retrieval-only reuse, so a harness state trained against one model can be carried over to another, though the paper does not claim that transfer preserves the full lift.

The evaluation uses eight interactive environments derived from τ²-Bench and MultiWOZ-2.4, customer-service-style and multi-turn dialogue benchmarks used to measure agent reliability. The reported number is a first-try task success rate, Pass@1, the share of tasks the agent solves on the first attempt without retries. Against the strongest interactive baseline the paper identifies, the layer adds 10.07 percentage points on τ²-Bench and 9.91 percentage points on MultiWOZ-2.4, an average lift close to 10 points across both benchmark families.

A 10-point jump with the model unchanged is a meaningful claim, so the methodology the number rests on is the part that has to hold up. That is where the companion paper, "Rethinking the Evaluation of Harness Evolution for Agents", enters the picture. That work argues that the way harness-evolution methods are scored is itself an open problem, and that a layer can post large deltas on a chosen harness-evolution benchmark without proving it would help in a real deployment.

The paper's "evolution" is procedural memory and state-graph updates, not model retraining. The authors are explicit on this point. The paper is best read as a constructive proof that a frozen model plus an accumulating repair log can move first-try reliability on synthetic, benchmark-style tasks, not a deployment-scale reliability claim.

The mechanism is reproducible. Anyone with access to a model and a task harness can wire in an Evolution-SOP and a state graph, freeze the rest, and try to reproduce the lift. The 10-point headline number is on the authors' own evaluation against the authors' own baselines, and the methodology that gives it meaning is now being argued over in the open literature.

The team does not describe a production system, a vendor launch, or a consumer product. The paper proposes a layer, names it, measures it, and invites replication. Whether the layer becomes a real lever for agent reliability depends on whether independent teams, on independent benchmarks, can recover the lift, and on whether the field settles the question of how harness evolution should be evaluated in the first place.