The author of a newly released open source fine tuning tool invited the community to attack its weakest piece, and a Reddit critique accepted. The exchange points at a structural gap in how the field measures catastrophic forgetting.
Pyrecall, a newly released MIT-licensed tool for catching catastrophic forgetting during LLM fine-tuning, comes with an unusual property: its own creator is the first to admit the benchmarks shipped in v0.1.0 are the weakest part. The announcement on r/MachineLearning explicitly invites the community to make them better, and a critical comment on the same thread from user marr75 accepted the invitation within a day. Read together, they sketch a more interesting story than either the launch or the pushback alone: the tool is real, the gap it tries to fill is real, and the benchmarks in this kind of tooling trivialize the problem they claim to solve.
Pyrecall does three things well enough to be useful. It takes skill-score snapshots before and after a fine-tuning run, flags regressions against a baseline, and rolls back LoRA adapters by name. The whole thing runs locally with no external API calls, and the GitHub repository is pip-installable. For a small team that needs a fast "did I break something" signal, that combination is genuinely missing from the standard fine-tuning stack.
That is also where v0.1.0's weakness lives. Pre/post snapshot scores are useful when the task being measured is the task being fine-tuned. They are less useful when the fine-tune drags the model off a local minimum in capability space that the snapshot was never going to detect in the first place.
The Reddit critique accepts that a snapshot can catch surface-level regressions, then makes a sharper claim: short, single-pass benchmark suites are not just blind to the hardest forgetting, they are actively dangerous when paired with a rollback button. marr75's argument, taken in good faith, is that a tight loop of "score dropped, roll back" can converge on a fine-tune that looks stable on the benchmark while the model quietly loses subtler capabilities that the eval never asked about. Rollback is not a safety net in that regime. It is a way of freezing the model at whichever configuration happens to score highest on the wrong test.
The critique's most useful move is naming the capabilities that short-form benchmarks structurally under-measure: multi-step planning, 2D and 3D geometric reasoning, non-English-to-non-English translation, 10th-grade writing, ethical-dilemma reasoning, and security or legal domain alignment. None of those show up reliably in a five-task pre/post suite. Each of them is exactly the kind of capability a practitioner might be most worried about silently degrading while they watch a benchmark number stay flat.
This is also why the v0.1 critique is interesting on its own terms. The Pyrecall author flagged the benchmark as the part they were least confident about, which is a more honest launch posture than the field usually gets. The community response did not punch down on the implementation. It used the invitation to point at the harder design problem.
The harder design problem, named directly, is the gap between what catastrophic forgetting looks like in production and what a short benchmark can detect. A capability-faithful evaluation suite for fine-tuning would have to do at least three things the current v0.1.0 does not attempt. It would have to sample across capability categories the fine-tune was never explicitly optimizing, in the same languages and formats the deployed model will see. It would have to distinguish between local regressions and drift on structurally different tasks, so rollback does not flatten legitimate learning in service of a narrow score. And it would have to keep those tests cheap enough to run on every fine-tune, which is part of why no one has shipped a credible version of this yet.
That last point is the one the Pyrecall launch quietly underlines. The tool exists because the community needed something. The benchmarks that ship with the tool are honest placeholders. The critique is constructive because the gap is well-defined and the author asked for help defining it. Whether Pyrecall grows into a more capability-faithful evaluator, or whether it stays a rollback helper with thin evals bolted on, is now a question about who shows up to do the work, not a question about whether the problem is real.