AIANALYSISPODCAST
'Token maxing,' as Penn describes it, is the closed loop of evals, runs, and scores that lets one builder ship what used to require hundreds, and the loop has limits she is candid about.
A small product team at Anthropic writes the tests that decide whether each Claude release is ready. The discipline, as she calls it, is "token maxing": build the eval, run the model, score the output, ship the better version, and start again. In a recent episode of Lenny's Newsletter podcast she argued, in her framing, that the loop lets a single builder do the work of a 400-person team, with the trade being headcount for judgment.
She joined in 2023 as Anthropic's first technical product manager, when the entire product team was five engineers, and her first job was to make the model-release process legible: what changed, what got better, what regressed. She says she has helped ship every Claude release from Claude 2 through Fable, the most recent model she names in the conversation, and has helped incubate Claude Code, the Model Context Protocol (MCP, the standard Claude uses to talk to outside tools), Skills (packaged prompt-and-tool bundles that give Claude a repeatable procedure), computer use (letting the model drive a desktop the way a person would), tool use, and the reasoning features that now ship by default.
The eval set is the operating asset, not a check at the end. The team writes the test suite for each capability before the model does the work, scores the output on a rubric the team agrees on in advance, and only lets a model graduate to the next task if it passes the previous one. The cycle repeats until the model stops improving on the metrics that matter, and the loop is what she means by token maxing: not a one-shot prompt tweak, but a closed system where the score is the spec. A new model that breaks an existing eval, in her telling, is not a release at all; it is a regression in disguise.
This is the part the rest of the industry is starting to copy. Outside labs, she argues, the bottleneck for shipping software is headcount; at Anthropic, the bottleneck is judgment. If a team can specify the test, they can run it on every model that comes out, including models from other labs, and the only thing the team has to scale is the quality of the rubric. Y Combinator's recent "Tokenmaxxing" library entry treats the same idea as a pattern founders can apply without working at a frontier lab. The eval loop is the part that travels.
The jagged edge is where the loop breaks, and she is candid about the limit. She uses the phrase to describe the parts of a task where a model looks superhuman on one input and clearly wrong on the next, the edges of capability that no eval fully captures. The eval-driven loop does not retire human judgment; it just moves it. The team still has to choose what to measure, and the choice is itself a product decision with its own failure modes. A rubric that rewards the wrong behavior scales the wrong behavior, and an eval set that stops discriminating between releases is a tax on the next round of work. The lab's own Anthropic Labs announcement and its research on Scaling Laws and Interpretability of Learning from Repeated Data are the receipts that the same method scales from a research artifact to a product surface.
For a non-Anthropic team, the prerequisite is not a frontier model. It is a rubric specific enough that two readers would score the same model output the same way, and a process for retiring evals that have stopped discriminating between releases. She is also clear that the practice she describes is Anthropic's operating discipline, not a recipe for any team that buys an API key next quarter. The way she tells it, the bottleneck is the discipline, not the model, and that is the part a builder outside the lab can actually try to copy.