Eight thousand queries is enough to clone a transformer's inner wiring, researchers find

A team of researchers has shown that fewer than 9,000 carefully chosen inputs can recover the hidden internal wiring of a transformer-based AI model, the kind behind services from image classifiers to ChatGPT-style assistants. The work, posted to arXiv this week, describes a new class of black-box model-extraction attack against hosted AI services.

"Black box" in this context means a model sold without exposing its weights. The attack targets the feed-forward network, or FFN, the two-layer block of weights that sits inside every transformer layer and does most of the per-token computation. Using only a chosen-input raw-output oracle (the attacker can feed the model any input and read back full numerical outputs, not just a top label), the researchers probe what happens to those outputs as the input is nudged. The probe reads curvature rather than answers: the rate at which small input changes bend the output surface, captured as projected input Hessians.

On independently trained CIFAR-10 vision transformers, the technique needed only 16 such projected Hessians, totaling 8,193 queries, to recover the FFN's hidden input-weight directions with average absolute cosine alignment above 0.94. By the authors' threshold breakdown, 95.1% of GELU-activated directions and 91.9% of SiLU-activated directions crossed 0.90 alignment. Recovery held across repeated runs, across independently trained models, and across all transformer blocks in the network, per the preprint.

The mechanism: the projected input Hessians are not raw second derivatives; they are mixtures of the same hidden symmetric rank-one factors induced by the FFN's input weights. The authors formalize this as a partially symmetric decomposition, which gives local identifiability (the hidden directions are recoverable up to a sign) and stability under small perturbations. A second move, vector-output stencil reuse, cuts the structural query budget by 16x relative to a naive scalar stencil. Together these give an end-to-end path from black-box second-order observations to recovered FFN directions and, downstream, to functional substitutes.

Functionally, holding the recovered directions fixed and fitting only the remaining FFN parameters, the authors built substitute networks that agreed with the targets on more than 93% of top-1 predictions, with test accuracy within 0.90% of the GELU target and 0.62% of the SiLU target. That is not a weight-for-weight copy. It is a behaviorally faithful clone, built from queries alone, the authors report.

The defenses the paper actually tests are narrow: output rounding and additive Gaussian noise. Both meaningfully degrade recovery under a fixed attack configuration. The catch is that the attacker can adapt. Re-tuning the finite-difference step used in the probe, the same step the defenses were tuned against, restored average alignment to 0.9603 on GELU and 0.9398 on SiLU. The paper's countermeasure analysis stops at output-level defenses. Gradient- or activation-level mitigations and architectural changes are out of scope in the abstract.

Three limits keep this from being a live exploit against the major hosted AI services. First, the demonstration is on CIFAR-10 vision transformers, two layers deep, with either GELU or SiLU activations; the paper does not claim, and the abstract does not show, that the same probe recovers frontier-scale language model internals. Second, commercial inference APIs typically do not return full per-branch numerical outputs to a chosen-input oracle, and many already apply output rounding, top-k truncation, or noise as part of their serving stack. Third, the work is an unreplicated arXiv preprint. Claims of robustness and effect size are paper-internal until another group reruns the experiment.

Output rounding and calibrated Gaussian noise degrade the fixed-config attack, and a defense that holds even briefly buys detection time, since any query pattern strong enough to recover FFN directions is also a query pattern that can be rate-limited or alerted on. The hardening that works against the attack must be re-evaluated as attackers tune the finite-difference step. The chosen-input raw-output oracle is the load-bearing assumption, which makes API surface design, not model architecture, the first thing to harden.

The next test worth watching is the one the paper does not run: whether the same probe survives a frontier-scale language model, a commercial-style output pipeline, and a replication team that did not write the original code.