AINEWS
LoKiFormer, a proposed model architecture described in an unreviewed arxiv research paper, separates attention from knowledge storage and reports 1.33× faster pretraining — the initial large scale training phase before a model is specialized.
A new arxiv preprint, LoKiFormer, argues that the next big lever in pretraining efficiency isn't more parameters or more data. It's putting an LLM's world knowledge somewhere else entirely, in a dedicated addressable memory, so the attention layers can spend their work on the local context each token actually needs.
The paper is a research proposal, not a product. It is not peer reviewed, and no independent lab has yet tried to reproduce its result. What it offers is an architectural idea with a number attached.
The premise starts from two complaints about how today's large language models are built. Every modern LLM is a transformer, a stack of layers that each read the previous layer's output and decide which earlier words to pay attention to when predicting the next one. The first complaint is that attention has no built-in sense of proximity. A word three positions back and a word three hundred positions back are processed with the same machinery, so the model re-derives the same local grammar and phrasing at every layer. The second complaint is about how world knowledge is stored. The current popular answer is mixture-of-experts, or MoE. In that design, each token activates only a subset of the model's "expert" sub-networks, and the model's facts live tangled up inside those experts. The authors argue that tangling makes it hard to update or retrieve a single fact without rerunning the whole computation.
LoKiFormer's fix is to split those jobs in two. The first piece, called Local Fusion Attention, adds a small convolutional step: a fixed-size sliding window of nearby tokens that gets folded into the attention calculation, so each layer has explicit local context to work with. The second piece, called the Knowledge Memory Module, is a separate bank of addressable slots, each slot a small learned vector the model can write to and read from. Think of it as a scratch dictionary the rest of the model can look up. Facts go in one place, and attention can stay focused on what's happening in the sequence at hand.
That separation is the architectural claim. The number the paper hangs on it is 1.33× faster pretraining convergence, meaning the model reaches a given training loss on roughly three-quarters the tokens of the baseline the authors chose. "Pretraining convergence" here is the count of training tokens the model has to read before it hits a target score, not wall-clock time or dollars. The number is real, and it is theirs. The abstract does not name the baseline model, the training dataset, the parameter count, or the compute budget, and the comparison runs on the authors' own setup. Without those details, "33% faster" is a claim worth recording, not a benchmark worth quoting.
What the paper does not yet show is whether the trick survives outside that setup. There is no downstream task quality table in the abstract, no standard benchmark, no human-eval score, and no statement about how the Knowledge Memory Module behaves when it is scaled up or fine-tuned on a specific domain. There is also no head-to-head against the current production alternative a lab would actually choose, which is a well-tuned MoE. Until those answers land, the 1.33× is a directional signal from one group, not a field result.
The constructive read is narrower than "AI just got 33% faster." It is that one research group is betting the cheapest path to cheaper pretraining is structural: keep attention local, keep knowledge stored, and stop paying for both at every layer. The full paper, and a replication, will decide whether that bet pays off.