New AI audit surfaces rare harmful replies and reverses some safety rankings

In a new preprint, a training-free audit pipeline called BLOOM-WILT lifts the rate at which one tested model, Qwen3.5-4B, can be probed into producing self-harm encouragement from 51% to 100%, with no extra training cost. The paper's authors argue the gain reflects test strength, because the model itself is unchanged across the comparison.

The technique is called logit tilting, and the doorway is a language model's own probability math. When a model picks the next word, it samples from a distribution over its vocabulary. Some rare-but-possible replies, the kind an auditor wants to surface, sit just below the sampling threshold the model normally uses. WILT nudges that threshold by reweighting the target's next-token distribution under an elicitation prompt, so behavior-relevant generations are sampled ahead of equally probable alternatives the model would have ignored. The paper's full writeup describes this as surfacing latent behavior rather than amplifying improbable outputs: behavior presence rises without pushing the model's output probability below the baseline's.

BLOOM-WILT is the pipeline around that trick. The "BLOOM" side is an auditor model that revises its conversational strategy across rounds, learning from previously scored interactions, so each probe starts from a sharper place than the last. The "WILT" side, the logit tilt, sits on the output. The whole pipeline needs only next-token access to the target, no fine-tuning, and runs at the same compute budget the authors allocate to their baselines. In 32 head-to-head settings across 4 target models and 8 behaviors, BLOOM-WILT beats the baseline auditor 30 times, and at matched compute beats every elicitation method the authors port into the same pipeline. That matched-compute framing is the meaningful one, because it controls for the obvious counter: that the method only wins because it spent more cycles on the probe.

That is the part that reframes the rankings. A stronger probe surfaces more of the behavior a model can already produce, so a model that looks "safer" on a weak test can land lower on a stronger one. The paper reports exactly that flip in its abstract and treats the ranking reversal as a finding about the test, not the deployment. The 51% to 100% number on self-harm encouragement for Qwen3.5-4B is the cleanest illustration, because the gap between "rarely elicited" and "always elicited" is the gap between a published safety score and a re-scored one.

The scope is narrow and the paper says so. The eight behaviors are the authors' selection; the four target models are the ones the authors could access at the needed logit level. The matched-compute claim is paper-scoped because it only covers the elicitation methods the authors re-implemented, and the 100% figure is reported on a single behavior and a single model. The abstract frames behavior presence as how often a probe elicits the behavior, not how often a real user would encounter it. In practice, the next-token access the method needs means more than a chat interface; it requires raw token-level output distributions, which most commercial deployments do not expose. The paper is also a preprint, with code, a transcripts dataset, and an Inspect extension released alongside it. The community has not yet audited which prior rankings the paper actually overturns, and the per-behavior results table is what a deployer would want to read before treating the reversal as a verdict on a specific product.

What changes for an auditor or a deployer is the unit of work. Probing no longer requires retraining or a fine-tuned red-team model; it requires logit-level access and a willingness to tilt the sampling math. The remaining question is whether published safety scores, which have generally been measured against weaker elicitation, will be reissued under stronger ones, and whether the eight behaviors the authors picked are the ones that matter most. The cleanest signal that the field has absorbed the result would be the first re-scored safety leaderboard run under logit tilting. The paper is a tool for that re-measurement, not a verdict on any commercial deployment.