What actually keeps a language model's memories from overwriting each other

A controlled experiment on two small language models (LMs) finds that swapping in a "digital" memory doesn't help them remember longer. A single-author preprint, posted to r/MachineLearning (self-post with full mechanistic summary) and hosted on Zenodo (record 22071715, DOI 10.5281/zenodo.22071715), compared a binary edge-mask memory adapter called EPMem against an architecture-matched continuous-weight adapter on TinyStories-33M and Pythia-410M, with storage bits and write steps matched across the two.

The binary version writes new facts fine, but forgets them at least as fast as the continuous adapter and underperforms it overall. The mechanistic check shows why: per-write interference scales as the square root of mask flips, and the training gradient preferentially rewires edges that already store something. The author calls that "anti-consolidation, not neutral overwrite."

A chunk of the continual-learning literature has bet on discreteness as a forgetting fix. Methods like SupSup and WSN avoid forgetting, the paper argues, because they enforce non-overlapping allocation between writes, not because the memory itself is binary. The allocation rule is the lever; the substrate is not.

The author flags a real caveat: the result may be partly an artifact of straight-through estimation, the gradient trick used to train the binary mask, not of discreteness per se. Scope is narrow: fact-writing on two small LMs (33M and 410M parameters), no independent replication. Generalization to larger models or other task families is not established.