Dan Luu's 26-prompt AI coding benchmark: most testing instructions did not reliably help

Dan Luu, a software-engineering writer known for measurement-driven essays, just published a benchmark testing whether AI coding agents actually benefit from being told to use specific testing techniques. The setup ran 26 prompt conditions on a real Rust implementation of the Zstd compression spec, re-using the eval Luu previously used to compare agentic programming languages.

The conditions ranged from process-flavored instructions like TDD, fuzzing, and property-based testing to heavyweight formal methods such as Lean 4 and TLA+. Luu also tested four pre-built "skills," including a Hegel skill, Trail of Bits' property-test skill, and a custom skill he wrote himself. The eval was framed as novice-guided: an instruction source with no testing expertise choosing techniques they have heard of.

Most of the 26 instructions did not reliably move correctness. Luu had pre-registered that TDD would underperform (55% confidence) and that formal methods would not overperform (52% confidence), and he flagged his own bias toward expecting TDD to lose. The post's framing thesis: it is easier than ever to hit a particular quality bar, yet software quality is getting worse, which suggests the defaults developers are using may not work very well either.

The source excerpt does not include the condition-by-condition pass-rate data, and a Hacker News commenter raised a methodological caveat: agents may not have actually used TDD or mutation testing as instructed, and the mutation testing may have been manual rather than framework-automated. Anyone treating the result as a verdict on AI code quality should wait for the full numbers.