AIBRIEF
A controlled study finds the math used to score words inside language models decides whether a learned memory deletion routine carries over from training to deployment. Scope: an earlier OpenAI language model trained on a public web text corpus.
When an AI lab trains a model to shed its working memory, the training process uses a soft dial: entries are attenuated, not removed. At deployment, that dial has to land on a hard wall. Only physically deleted entries free up GPU memory. A new preprint documents a hidden mismatch between the two regimes, and shows the choice of attention math, sigmoid vs softmax, decides whether the soft dial ever reaches the hard wall.
The paper runs a controlled 2×2×2 design across attention type, learned gating, and positional encoding, using GPT-2-scale Transformers trained on OpenWebText. Sigmoid attention is worse as a dense language model on its own. Paired with learned hard eviction, it deletes KV entries with negligible change in perplexity relative to its own no-eviction baseline. Under a matched live-cache protocol, learned sigmoid gates also beat the authors' re-implementations of H2O and KeyDiff. Softmax gates do not uniformly outperform those post-hoc baselines.
Scope: GPT-2 on OpenWebText, preprint not yet peer-reviewed, and the H2O and KeyDiff comparisons are the authors' own reimplementations rather than the strongest current post-hoc baselines. Scaling to production-size models is not tested here.