AIBRIEF
Tensormesh and AMD say offloading AI memory to DRAM and SSDs on a Dell server with 8 AMD MI355 GPUs cut time to first token (the wait before a model starts replying) from 3.4s to under 0.5s and roughly doubled models served.
Tensormesh and AMD published joint benchmark results showing that an AI server can roughly double the number of models it runs by moving the model's working memory off the GPU and onto cheaper server storage. The test ran on a Dell server with eight AMD MI355 GPUs against a 300GB document workload using the Kimi-K2.6 language model.
Time-to-first-token, the delay before a model starts replying, fell from 3.4 seconds to under half a second. The system reused cached data in DRAM and network file storage instead of recomputing it. Output held at about 48 tokens per second as the workload grew, while an unoptimized baseline dropped roughly 40%. Model density, the count of distinct models a single box can host, doubled (Tensormesh blog; AMD press kit).
Tensormesh's LMCache and AMD's virtual-memory tools did the work. The LMCache layer stores the "KV cache," the intermediate attention data an AI model produces as it reads a prompt, so the model can reuse that data on the next request. AMD's virtual-memory tools let that cache spill across DRAM, SSDs, and remote storage instead of staying pinned to scarce on-chip high-bandwidth memory, or HBM (LMCache benchmark on AMD MI300X).
The numbers are vendor-supplied and use one model family. AMD Ventures has a financial stake: it invested in Tensormesh's USD $20 million round (ChannelLife; ITBrief NZ; VMblog). Independent reproduction on other models and production traffic remains open.