Researchers Used an LLM to Map Another LLM's Internals

An arXiv preprint released this week reports a concrete win in circuit tracing, the technique for reverse-engineering what a language model actually does when it answers a prompt. On a standard two-hop Capitals probe, a pipeline that uses one language model to label the internal features of another recovered the correct intermediate step in 97 of 100 prompts.

Circuit tracing follows which internal features of a model fire on which inputs, then groups those features into "supernodes," the meaningful clusters a human would have labeled manually. The grouping is what makes the technique legible. Without it, researchers get a forest of individual neurons and activation patterns with no map. With it, they get a readable account of which features the model used to reach an answer and which it ignored.

The bottleneck has always been that last step. A trained annotator has to look at hundreds of features, read what each one responds to, and decide which belong together. A single circuit can take days. That is fine for a flagship paper. It caps what a small research team can audit in a given week.

The new pipeline, "LLMs Can Annotate Attribution Graphs," hands the grouping step to a language model. The model is shown descriptions of each feature and asked which belong in the same supernode. The output is a labeled attribution graph ready for human review.

The two-hop Capitals task asks a model to chain through an intermediate fact, like naming the capital of a country reached only by first identifying its largest city. The pipeline has to surface the supernode corresponding to that intermediate step. Recovering it in 97 of 100 prompts is the concrete signal that the grouping works. Using automated interpretability metrics, the authors report that the labels it produces are as interpretable as labels written by a human annotator. (arXiv:2608.02632)

The same pipeline automatically annotated 1,000 attribution graphs drawn from Wikipedia prompt completions. A separate language-model judge then flagged the interesting ones for human review, so a researcher does not have to triage the full thousand. The companion code is on GitHub at maxh119Z/circuit-tracer-automation.

The result is not a claim that anyone now "understands" large language models. The technique still only covers narrow behaviors, and the source is an arXiv preprint, not a peer-reviewed paper. The 97-of-100 result is on a specific two-hop task; broader generalization is not shown. Author names and affiliations beyond a single GitHub handle are not visible in the abstract. (arXiv html)

That shifts the rate-limiter. The most labor-intensive step in the field's most promising interpretability technique moved from expert hours to compute. A small team can now run the same audit a top lab could afford last quarter. The next legible claim about what a model is doing inside will likely come from a smaller group, on a narrower question, audited faster. That is how this corner of the field expands: not with a single breakthrough, but with the slow compounding of what a week of research can cover.