The 'Judges' Training AI Assistants Are Opaque. A New Paper Tries to Read Them.

Every modern chatbot was shaped, in part, by an internal "judge" that compared two of its answers and picked the better one. Those judges are themselves largely opaque. A new paper argues we can read them more honestly: by listening to how they actually weigh each response, not just which expert they route it to.

The judge in question is called a reward model, and it sits inside the training loop, not in front of a user. When a lab trains an assistant to be helpful, harmless, and coherent, the lab first generates pairs of candidate responses to the same prompt. The reward model scores each pair, and the training signal is whatever the model prefers. A chatbot that answers email, summarizes a meeting, or refactors code is, in a real sense, the residue of millions of those pairwise judgments.

The problem is that no one outside the lab can see inside that scoring model. It is a neural network of its own, often larger than the model it is shaping, and it is treated as a black box even by the people who build it. A new arXiv preprint from researchers working on AI interpretability, "Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast," proposes a specific tool for reading one increasingly popular family of these judges: the Mixture-of-Experts, or MoE, reward model.

A Mixture-of-Experts model breaks a neural network into many small "expert" sub-networks and adds a router that sends each input to the experts best suited for it. The same architecture shows up in many large language models, where it lets a network have more parameters without paying the full inference cost on every input. When applied to reward modeling, the result is a judge made of many specialized experts, each one trained to score a different slice of the response space.

The standard way to peek inside such a model is to watch the router. Which expert is being asked to judge this response? But the paper's authors argue that routing weights only show which prompts an expert sees. They do not show how the expert judges the response once it is selected. Two responses sent to the same expert can produce very different scores, and the routing data alone cannot tell you why.

Their proposed fix is called CoCo, short for Contribution-Contrast. Instead of watching which expert fires, CoCo looks at pairs of responses, one chosen and one rejected, and finds the expert whose contribution to the score changes the most between the two. The experts with the largest contribution gaps are the ones doing the actual judging work. By reading those experts in the context of the responses they preferred and rejected, CoCo is meant to produce a characterization of what each expert is really for.

The paper's authors position this as the first systematic attempt to interpret MoE reward models at the response level, and they report that CoCo produces more coherent, faithful, and specialized explanations than three baselines: a router-only analysis, a generic score-based analysis, and an analysis built on sparse autoencoders. They also report that the method does not collapse the judge's accuracy. A reward model inspected with CoCo still scores chosen-versus-rejected pairs about as well as before.

The honest limits are visible in the source itself. The paper is a single arXiv preprint, not a peer-reviewed study, and its evaluations are run on the authors' own benchmarks. The authors' "first systematic study" claim is self-attested. CoCo is a methodology contribution evaluated in a controlled setting. It is not a safety fix, not a deployment change, and not a claim about any shipped chatbot. A user talking to a deployed assistant will see nothing different.

What the paper does add is a sharper mental model of where AI opacity actually sits. The chatbot a person talks to is one layer. The judge that scores training responses is another, structurally similar, often larger, and equally opaque. The interpretability project, the cumulative field-wide effort to read these internal models, is how the community builds shared evidence about what its own systems are doing. CoCo is one specific tool in that project. It is a way of looking at which experts inside a MoE reward model are actually pulling the lever on each decision, rather than which ones happened to be on shift.

The test for any new interpretability method is whether it survives contact with the larger, messier reward models actually used to train shipped systems. CoCo's authors will need to show that the contribution-contrasts they measure are stable across training runs, model sizes, and the more aggressive routing smoothing that production-grade MoE systems tend to use.