A researcher asked whether an AI can be caught before it lies. The honest answer across 10 models: not with one detector, but with a baseline.

A researcher published a pre-registered experiment in late July to test whether an AI model can be caught hallucinating at the moment it commits to its first word, before it has said anything wrong. Across 10 models, the honest result is no universal detector and a drop-in score of 6 of 10. A baseline does hold up, and every number, every signal, and a verification script anyone can run without a GPU are public, so the field can falsify it.

"Hallucination" is the field's term for an AI confidently stating something false as a fact. "Pre-registered" means the researcher locked in the analysis plan and the success criteria publicly, before the data existed, so the result could not be reverse-engineered to look good. Michael S. R. Kitti, a single author working under a pseudonym with a Proton address, posted the plan and the paper to Reddit's r/MachineLearning on July 26, 2026, and hosted the paper, under a Creative Commons license, on his own site furnace.baby/cc-paper. The code is on GitHub.

Existing detectors work after the fact: a model has already produced text, and a second model judges whether the text is true. Kitti's bet is that the commit moment, before any text is shown, carries the answer in the model's internal geometry. A "frozen model" is one whose internal weights are not being updated during the test, so any read of its state reflects the model as it would actually be deployed. The test runs in a single "forward pass," one full read-through of the input, and produces 29 internal signals across four families: attention morphology (the shape of where the model looks), residual-stream motion (how the internal representation changes as it propagates through the layers), readout geometry (the geometric arrangement of the final internal state), and the model's own confidence. Selection of which signals matter runs on a nested out-of-bag bootstrap with sign-locking per deployment.

Run 1 covered two tasks and 10 models, for 20 deployments. A geometry-only detector cleared the pre-registered bar at 18 of 20, against a threshold of 17. Adding the model's own confidence left the count at 18 of 20, with the same two misses and zero rescues, so the stricter pre-registered claim that "confidence covers more" failed. Confidence is redundant with geometry on this panel. Twelve different signals win across the 18 working cases. The top pick is per model and per task.

One fixed signal combination, calibrated on 9 models and tested on the held-out 10th, then rotated through all 10, still beats chance on 9 of 10 cases on ANLI (a natural-language inference benchmark where the model decides whether one sentence follows from another) and 10 of 10 on TriviaQA (a public question-answering set of trivia questions). Run 2 extended the panel to six tasks, including HaluEval-QA, a benchmark built around hallucinated answers. Per-model calibration hits 10 of 10 above chance on the new task. The one drop-in detector Kitti pre-registered, fixed signal, fixed sign, applied blind, lands at 6 of 10. The four misses do not lose the signal, they read it backwards, with AUROC (area under the receiver operating curve, where 1.0 is perfect and 0.5 is chance) falling as low as 0.17.

There is a check on whether the result is just a numerical artifact. The signal was re-tested across four precision formats that change how the model stores its math: nf4, int8, bf16, and fp32. On the models where the detector works, the result is precision-invariant. Int8 wobbles on small models, but the wobble washes out by 32 billion parameters. The signal is measuring real computation, not rounding noise.

Kitti has chosen to make the open question public. He released the full score matrices and a verification script, stage_b/verify_endpoints.py, that re-derives both pre-registered verdicts without re-running the models. No GPU required. "Come break it," reads the post title. If independent replication finds the inverted-read pattern on held-out models, the floor does not generalize and the result collapses into a per-model artifact. If the floor holds, the field has a baseline it can build on, and a public, falsifiable test for whether any future claim of "we caught the lie before it happened" actually does.

Kitti's paper sits next to a small but growing body of work on catching hallucinations from a model's internal state, including a recent arXiv preprint, Layer-wise Semantic Dynamics, that reports an AUROC of 0.96 and an F1 of 0.92 (a precision/recall balance score where 1.0 is perfect) on TruthfulQA from a single forward pass via hidden-state geometry. Pre-registration is what turns a number like that into something the field can rule on. Kitti published a plan, a held-out test, an honest 6 of 10, and a script anyone can run to check the score.