How AI products could keep long-term memory without burning the GPU budget

When an AI assistant remembers you, every new question has to start with the system re-feeding the model your accumulated history before it can answer. That prefilling step is the hidden reason a memory-heavy product feels slow, and it is also the reason serving one of these products at scale gets expensive: response time and compute cost grow with how much the assistant knows.

A new arXiv preprint, InferScale: GPU-Native KV Injection for Personalized LLM Serving, attacks that bottleneck by reusing the model's already-computed internal state instead of recomputing it on every request. The paper reports a 72% to 79% cut in time-to-first-token (the gap between sending a question and getting the first word back) at a retrieval budget of 50 memory facts, with throughput (the number of requests a single GPU can serve at once) up 3.7x to 4.5x under concurrent load.

The mechanism is narrower and more specific than the headline number suggests. Modern LLMs store intermediate values from earlier layers in a key-value cache, the working scratchpad the model consults instead of re-reading every previous token. Today, the standard way to give a model memory is retrieval-and-inject: retrieve relevant facts, paste them into the prompt, and let the model recompute the cache for the whole enlarged prompt. That is what scales badly. InferScale instead precomputes each memory fact's cache, stores it alongside a semantic embedding (a vector representation of meaning used to find the right fact) on the GPU, and at serving time splices those stored caches directly into vLLM's paged cache (vLLM is a popular open-source LLM serving engine; a paged cache is its method of splitting the cache into addressable blocks) through the engine's KV-connector interface. No model retraining, no engine fork.

Two pieces of plumbing make the splice work. The first is Chunked RoPE: rotary position embeddings (RoPE) are how most modern LLMs encode the position of each token, and the usual approach breaks if a cached fact was computed at one position and is later injected at a different one. InferScale stores keys before the position rotation is applied and re-applies positions at serving time, so cached memories stay positionally valid wherever they land. The second is Context-Window Encoding: each fact is encoded with a small window of preceding conversation context, but only the target fact's cache is stored, recovering some of the cross-fact context that independent encoding loses.

The integration surface is the load-bearing constraint. InferScale ships as a pure vLLM KV-connector v1 module implementation, which is why the design is feasible at all. vLLM is the serving engine most production stacks already run, and its KV-connector interface is the documented seam for plugging in cache-transfer behavior without forking the engine. That detail is what separates this paper from a generic make-the-cache-bigger optimization; the wins come from the seam, not the model.

The trade-off is honest and visible. On the LoCoMo long-conversation-memory benchmark (a public test for whether an assistant can recall facts across many turns), three open-weight models scored 60.3% accuracy with InferScale versus 63.3% for a Mem0 baseline that does no serving-time recomputation. That is a roughly 3-point gap, and the paper does not pretend it is free. The right way to read it: a small, named quality cost in exchange for a flatter serving curve as memory size grows. The comparison is also against a baseline that intentionally does no recomputation, so the gap is between two serving designs, not between serving and not-serving.

What this does not yet show is the part a serving engineer or product team would want next. The numbers are paper-claimed, not independently benchmarked, and the code release at saltsystemslab/InferScale is the project's first public artifact rather than a production track record. There is no customer evidence, no deployment footprint, and no cost study against real traffic. The paper-stage result tells a reader what the mechanism is and how it compares on a public benchmark; it does not yet tell them what it costs to run, what breaks at 10x the memory budget, or how it holds up outside LoCoMo.

For memory-bearing AI products, that distinction is the useful one. The model is not getting smarter in this paper. The serving stack is getting cheaper, and the mechanism that makes it cheaper is now small enough to plug into a standard engine without a fork. Whether that change is durable depends on results the paper has not yet measured.