A tiny on-device AI that calls for backup when it isn't sure

A small on-device model usually has two ways to handle a question it can't answer. It can guess, or it can shrug. Cactus, a startup run by Henry and Roman, has been demoing a third option on Hacker News: give the model a per-answer confidence score as structured data, and let it defer to a larger model only when the score says it should.

The technique is a probe that reads hidden-state activations across the layers of a post-trained checkpoint of Gemma 4 E2B, returning a single number between 0 and 1 alongside every answer. The score isn't parsed from the model's prose, and it isn't a string match on "I'm not sure." It is a separate signal, trained separately, designed to be calibrated enough to drive routing decisions.

If the score falls below 0.85, the prompt is handed to Gemini 3.1 Flash-Lite. If it clears the threshold, Gemma 4 E2B answers on its own. The whole stack, from probe to routing rule to the post-trained checkpoint, is open on GitHub, and the model weights sit on Hugging Face.

Cactus is not claiming to have built a self-aware model. The headline on the Show HN post says "we taught Gemma 4 to know when it's wrong," but that is a metaphor overstating the result. What the makers actually built is a small inference primitive, a confidence number a downstream router can act on, returned as structured data instead of embedded in the answer text. The user-visible behavior is the same as any confidence-routed hybrid, where a phone-sized model answers when it is sure and asks a bigger one when it isn't, but the routing decision is now a clean signal rather than a heuristic on the answer string.

The makers tested two alternative signals and found them weak. Text-based self-rating, the standard "on a scale of 1 to 10, how confident are you" trick, sat too close to the answer prose and tended to inherit its biases. Token-entropy heuristics, which score uncertainty by how spread the next-token probability is, were too noisy to use as a routing threshold. The hidden-state probe beat both on the makers' own benchmarks. None of those tests are published, so the comparison should be read as vendor-attributed.

The benchmark table the makers shipped tells a more honest story than the headline 15 to 35 percent handoff rate. At FP16, per-benchmark handoff rates range from 15 to 20 percent on ChartQA to 30 to 35 percent on MMBench, GigaSpeech, and MMAU, with LibriSpeech at 25 to 30 percent. MMLU-Pro is the obvious outlier, with 45 to 55 percent of queries still needing handoff to match Flash-Lite. If the goal is a small model that covers the long tail of hard reasoning tasks, the probe doesn't get there on its own.

Quantization makes the picture worse. The makers' own numbers: 4-bit adds roughly 10 to 15 percentage points of handoff to most benchmarks, and 3-bit breaks MMLU-Pro entirely. The repo's README flags that independent benchmarking of Unsloth, GGUF, and MLX quantizations is still needed. The numbers are not a clean "phone AI matches Gemini." They are a checkpoint- and bit-width-dependent demonstration of a routing technique.

So the news is a primitive, not a new model. A confidence probe that returns structured data, trained on a specific post-trained checkpoint, that lets a small on-device model defer to a larger one when the signal says it should. The Gemma 4 E2B numbers are one demonstration. The open question is whether the probe generalizes to other checkpoints, other bit widths, and prompts outside the test suite. Cactus has put a workable version of that primitive on the table, and whether it travels is the engineering work that comes next.