AINEWS
New data from the Terminal Bench 2.0 evaluation shows the harness iteration technique does not produce uniform gains — the per backbone differential is itself the follow on finding.
Three weeks ago a Shanghai AI Lab paper argued that the operating layer wrapped around an AI agent — the harness — is a first-class optimization surface, not just the model underneath it. The paper, Self-Harness, described a three-step closed loop: mine failure patterns from execution traces, propose bounded edits to a pre-declared harness surface, and keep the edit only if a held-in or held-out split improves without the other degrading. That initial coverage ran as a standalone explainer.
What the follow-on evaluation data adds is the per-backbone spread — and it is not uniform. On Qwen3.5-35B-A3B the held-out improvement was 104%. On GLM-5 it was 24%. On MiniMax M2.5 it was 28%. Same method, same benchmark (Terminal-Bench-2.0), same evaluation protocol held fixed. The three backbones did not converge.
The divergence is the story. The paper shows that each model's failure signature looks different. One explores too long without delivering artifacts. Another enters tool-failure retry loops that overwrite or delete required files. Self-Harness tailors its edits to those signatures, not to a generic agent profile. That is the mechanism answer, and it tracks: a model that consistently overruns its exploration budget responds to a different harness constraint than one that loops on tool errors.
The practical implication for teams evaluating agent tooling: the ceiling on harness-driven improvement is model-specific, not a uniform property of the technique. A 24% gain on GLM-5 and a 104% gain on Qwen3.5-35B-A3B are not the same signal, even if they come from the same pipeline. Teams on smaller or mid-range backbones should not read the Qwen headline and expect a comparable lift.
"Self-improvement" remains constrained to a pre-declared editable surface and gated by regression tests. Model weights are not modified. Terminal-Bench-2.0 is a coding and terminal-usage benchmark, not a general agent-deployment result. The code is public under qzzqzzb/Self-Harness.
Lilian Weng featured the work in a Harness Engineering for Self-Improvement post. Harrison Chase reshared it the same week.
The prior flagship piece, "The harness ate the model," ran on 2026-07-16T22:32:45Z and covered the Self-Harness mechanism in full. This piece is a follow-on data point on per-backbone differential, not a standalone re-report of the mechanism.