AINEWS
Kenton Varda's Team Drops AI PR Descriptions: Here's the Test That Reveals Why
One credible engineering voice calls out a failure mode most teams running AI summaries already recognize: descriptions that restate code rather than explain it.
One credible engineering voice calls out a failure mode most teams running AI summaries already recognize: descriptions that restate code rather than explain it.
Varda. The category covers pull-request summaries, commit messages, and the prose that ticks off issues and tickets.
His reason, surfaced by Simon Willison's Weblog, is a failure mode worth naming. AI-produced descriptions were bad in a specific, predictable way: they restated the code while skipping the framing a reviewer needs.
"AI was writing change descriptions that were worse than useless to me as I tried to review PRs: outlining details of the code that could easily be seen by looking at the code, but omitting the higher-level framing needed to understand broadly what the code is doing," Varda wrote.
A model that summarizes a diff sees the diff: function names, renamed variables, swapped conditions. It does not see why the change exists, which constraint it satisfies, what alternative was rejected, or what the reviewer should check next. It optimizes for what is locally visible in the patch and produces prose that looks like a description while draining the only thing the diff does not already contain.
Call the behavior "code-restatement optimization." Test it against your own toolchain: take the last AI-written PR description your team shipped and ask a non-author reviewer whether it told them anything they could not have learned by reading the diff in thirty seconds. If the answer is no, the description is not a description; it is an echo.
Varda's moratorium is not a product review or an industry verdict. It is one credible engineering voice declining to outsource the layer of writing that depends on intent the model cannot infer.
His team can keep using AI where the input is mechanical and the output is verifiable: code generation, test stubs, refactors with clear pass or fail criteria. The change descriptions stay human because writing them is the step in the loop where the author's reasoning has to land somewhere a reviewer can read.
The audit for every team that runs AI summaries in code review, incident postmortems, or release notes is the same: ask whether the prose told the reader anything they could not have gotten from the input in thirty seconds. If not, the prose restates. The description is then the part that costs the reviewer time, not the part that saves it.