Small AI agents trained from larger teachers over call tools; new research points to a handful of structural tokens, and a per token calibration cut that rate from 13.7% to 9.0% on APIGen MT.
When a distilled AI agent calls a tool instead of answering directly, the culprit may not be a bad example or a weak teacher. It may be a small set of structural tokens quietly steering the whole generation.
A new analysis of multi-teacher on-policy distillation shows that aggregate training signals can look healthy while the model drifts toward tool-overuse. The paper, titled "Behavior Leverage Imbalance in Multi-Teacher On-Policy Distillation," introduces a diagnostic for the failure mode and a calibration that addresses it without retraining the student.
Multi-teacher distillation trains a small "student" model from two or more specialist "teachers." In the agentic setting the paper studies, one teacher specializes in calling tools, another in answering directly. The student is then trained on its own generated tokens, a setup called on-policy distillation. The intuition is that the student absorbs the right mix of behaviors from the right specialist at the right moment.
Vanilla generalized knowledge distillation, or GKD, does improve tool-call recall. It also makes the student over-call. On the APIGen-MT benchmark, the distilled student reaches for tools on examples a human would answer directly, a failure mode the paper quantifies at 13.7%.
The interesting finding is what does not explain the failure. Tool-call samples do not receive more token exposure in the training mix. The full-sequence per-token divergence between student and tool-call teacher is no larger than for the answering teacher. Standard loss accounting sees nothing wrong.
The paper argues the gap hides in plain sight, inside the token-level mechanics of generation. A handful of structural tokens, including the <tool_call> tag and the function-name tokens that follow it, sit at mode-entry positions where the model decides which generation mode to enter. Mode-entry positions are unusual because a small probability mass on the right token, even with low confidence, can flip the whole trajectory from "answer directly" to "call a tool." Once the model has emitted a <tool_call> marker, the rest of the generation is constrained: the next tokens will be arguments, a closing tag, and a return to generation. The paper calls this phenomenon "behavior leverage imbalance": local signals at structural positions exert disproportionate control over the global generation mode.
The remedy is a per-token calibration called Soft Clamp. It caps the divergence contribution at structural tokens, where the leverage is highest, without retraining the rest of the model. On APIGen-MT, Soft Clamp reduces over-calling from 13.7% to 9.0% while keeping decision accuracy within the same range. The Berkeley Function-Calling Leaderboard multi-turn diagnostic, used as a secondary signal, points the same direction.
Soft Clamp is one calibration lever on a benchmark-grade failure mode. It does not claim to fix every reason a distilled agent over-calls, and the paper is explicit that aggregate loss and full-sequence divergence are still insufficient as sole signals. Builders get a named dimension to audit: behavior leverage at structural tokens, measured directly, not inferred from headline metrics.
For teams running on-policy distillation for tool use, the play is to track tool-call rate against a baseline of human-preferred behavior on the same distribution. If the rate climbs while aggregate loss and per-token divergence look stable, the failure is likely at the mode-entry tokens the paper identifies. A targeted per-token calibration is a cheaper intervention than a full retrain.
Beyond the benchmark, the diagnostic points to a broader pattern in agent reliability. When a distilled agent behaves badly, the failure is often blamed on the training data, the teacher mix, or the reward signal. The paper's evidence points elsewhere. The decisive moments are the few tokens at mode boundaries, and the right diagnostic is per-token, not aggregate. That distinction may matter less for the headline benchmark than for the next training run a team is about to launch.
The paper is a preprint, not peer-reviewed, and the Soft Clamp method section is truncated in the public summary. APIGen-MT and BFCL are public benchmarks, so the over-calling result is reproducible, and the diagnostic is the kind of finding that earns its place in the training loop the next time a team sizes up a distilled agentic model.