A new coding test asks AI to build software like a real engineer. The best models still can't.

A developer gave three of Anthropic's newest Claude models a six-hour coding exam this month. The test, SlopCodeBench, was built to mirror how real software work actually unfolds: build a feature, then adapt the codebase as new requirements land. The result was a small-subset pass rate of 24% for Opus 5, with the other two models clearing 6% each, and a verbosity curve that ran in the wrong direction across all three.

SlopCodeBench. Each checkpoint only reveals the next requirement after the prior one is solved, and the model has to keep earlier regression tests green. It is not a coding competition; it is a test of code stewardship, meaning whether a model can keep a codebase coherent over time rather than just emit plausible snippets in one shot.

The original arXiv paper found the best models at the time clearing just 14.8% of checkpoints in strict-pass mode. Verbosity rose 2.3x and structural erosion 2.0x compared to 473 open-source Python repos. Structural erosion, the measure of how much a codebase's shape degrades as it grows, showed up in 77% of trajectories; verbosity growth in 75.5%. The pattern was benchmark-level, not a quirk of one model's mistakes.

The independent scbench.ai leaderboard, which lets outside runners submit their own results, has since seen top entries climb to 28.1% (GPT 5.5/Codex), 26.0% (GPT 5.3-Codex), and 25.5% (GPT 5.4/Codex), with Claude's Opus 4.6 and 4.7 tied at 20.9%. The frontier is moving, but the bar is still low.

The most recent read comes from dex at HumanLayer, who ran Opus 4.8, Sonnet 5, and Opus 5 on a 3-problem / 17-checkpoint subset of the benchmark: circuit_eval (8 checkpoints, easy), database_migration (5, medium), and dynamic_config_service_api (4, hard). Fresh context per checkpoint, all in the Claude Code harness, with the original "just-solve" prompt. HumanLayer is an AI agent toolmaker; dex's framing is a builder's read, not an independent benchmark.

Opus 5 cleared 4 of 17 strict passes. Opus 4.8 and Sonnet 5 each cleared 1. None of the three runs reached checkpoint 8 on the "easy" circuit_eval problem. The 24% figure is a small subset chosen by one developer, with Claude itself picking the problems; it is not directly comparable to the 14.8% original-paper number or the 28.1% leaderboard top. It is a signal, not a score.

The pass rate is not the story. The code-quality metrics are. Across the 41 deterministic measures in the benchmark (the set of static checks that run without an LLM judge), slop-flagged line share averaged 98% for Opus 4.8, 93% for Opus 5, and 89% for Sonnet 5. Verbose-line share climbed from roughly 65% at checkpoint 1 to about 80% at checkpoint 8 for all three models. Opus 5 wrote approximately five times more functions and callables (named routines the code defines) than Opus 4.8 across the same challenges. The smaller model's real production volume was about 1.8 times the larger one's; the rest was expensive verbosity.

The one metric where Opus 5 looked better was duplication: clone-line share (the fraction of lines that are near-duplicates of earlier lines) moved from 2.41 to 2.64 across the run, against Opus 4.8's jump from 4.6% to 16.8%, with the inflection at checkpoint 3, where new requirements start fighting the original design. Sonnet 5 had 71.5% single-use functions versus Opus 4.8's roughly 50%. Mean cyclomatic complexity, a count of independent paths through a function, rose about 70% across checkpoints 1 through 8 for Opus 4.8, with the worst single function at cyclomatic complexity 93.

dex's read on the data: SlopCodeBench is unsaturated at the frontier. The strict-pass bar matches the question of whether a model can maintain a codebase over time, and today's models still cannot be trusted to run "lights-off" (a term for unsupervised, no-human-in-the-loop operation). His one-line summary: "every dollar bought correctness; nobody bought enough of it."

His proposed remedy is to amplify the signal: have a frontier model write checkpoints 1 through N, then drop to a smaller model (Sonnet 5 or Haiku) for checkpoint N+1. If the early code is well-factored, the smaller model can finish. If it is not, the failure becomes a fast, cheap test of the codebase's structure.

Practitioner reactions on the post were mixed. One user reported replacing Opus 4.8 at its highest reasoning effort with Opus 5 at medium effort and getting fewer tokens and faster runs. Another flagged weekly and daily drift in inference-cost tuning as a separate, ongoing problem. The discussion is color, not pass-rate evidence.

The autonomy narrative in the AI industry is outrunning the data. SlopCodeBench is the cleanest calibration update of the year, and it lands in the same place the original paper did: code generation is improving; code stewardship is not. For anyone shipping production software, the verbosity curve and the structural erosion are the read, not the headline pass rate. Watch the next leaderboard cycle; if the strict-pass bar moves above 35% on full-problem runs, the stewardship story starts to shift.