A new caching trick makes AI chatbots start replying 4.5 times faster

A caching technique called KVBoost cut the time an AI chatbot takes to start answering by 4.5 times in a new test — without retraining the underlying model.

On 1,000 bug-localization samples, the Qwen2.5-3B model took 142.4 milliseconds to produce its first token with KVBoost, versus 639.1 milliseconds without it. Accuracy was 99.2% with the cache active, compared to 99.1% without. The system also beat a common technique called prefix caching by 16% on the same test.

The gain comes from how language models reuse internal scratch work. When a model processes a long prompt, it stores intermediate calculations in a key-value cache that it can reuse on later requests. Existing systems only reuse that cache when two prompts start with the same words, because they key entries by position. KVBoost adds a second identifier — a content hash — so it can reuse cached chunks that appear anywhere in the prompt, not only at the beginning.

To keep that reuse from corrupting attention at chunk boundaries, the authors added two repair passes. SelectiveRecompute re-encodes a small boundary region; CacheBlendRecompute runs a fast probe pass and recomputes the tokens that diverge most. The system also supports int8 and int4 quantization, adaptive chunk sizes, and importance-weighted eviction under a fixed memory budget.

The preprint is RoPE-compatible — it works on the rotary position encodings most modern open models use — with no architectural change. The catch: the evaluation covers one 3-billion-parameter model on a single software-engineering task, with no independent replication yet.