SIGNAL
Personalization benchmarks are supposed to measure whether an AI has learned a user. The most popular way of teaching a frozen language model to adapt, by evolving a single shared prompt and then freezing it, may instead be measuring how cleanly the system can polish its instructions. The validation score the field has been optimizing cannot, in this setting, tell the right user's examples from the wrong ones.
On 200 held-out users per benchmark across two standard personalization tasks (LaMP-2 news tagging and LaMP-3 product rating), the meta-validation score was statistically indifferent to user identity: p=0.555 on LaMP-2 and p=0.622 on LaMP-3. The score the training process watches is blind to whether the prompt is actually using the right user, so it rewards better-written instructions and validation overfitting instead of adaptation. The authors call this meta-objective collapse.
The reading most coverage will reach is that Muse specifically failed. The reading the paper supports is broader. Any personalization claim whose only evidence is a benchmark lift, with no invariance test, is measuring polish. The paper's three-control protocol, which uses a seed prompt, wrong-support pairs, and an invariance oracle, is the reusable test. Personalization claims that survive it earn their keep; claims that do not are writing prompts, not learning users.
Every product demo that ranks assistants on benchmark lifts just lost a cheap tool for telling signal from dressing. The ladder was the score.