HWNEWS
As AI data centers hit rack power limits, the industry is rewriting efficiency around useful work per watt, a new north star for every AI infrastructure announcement that follows.
Some AI operators cannot run their GPU racks at full power because the building cannot supply enough electricity. That customer-side fact, not a chip-roadmap reveal, is the throughline of a recent round-table of six chip-industry executives convened by Semiconductor Engineering. It is why "power per token" (useful AI work delivered per watt) has become the industry's new north star, and why three engineering responses are already in motion.
A cluster is the unit above a rack: many racks of accelerators, network gear, and storage tied together with high-bandwidth fabric. It is one level up from the "GPU shortage" story the wire has been telling. At that scale, the binding constraint is no longer silicon. It is the substation feeding the building, the cooling loop carrying heat out, and the rack-level power-distribution units between them.
"Power consumption is a growing problem specifically with clusters," said Vishwakarma. "Customers are hitting rack power capacity limits and cannot run all GPUs at full power."
Cameron Brunner, head of HPC development at Siemens EDA, added that operators are responding by running accelerators at roughly half their rated power, not because the chips demand it, but because the racks will not deliver more. He noted that running GPUs at 50% power does not double completion time, since most jobs still finish within tolerable latency, and the energy-efficiency trade is favorable. The shorthand the panel kept returning to: power per token. Useful AI work per watt is becoming the metric that customers, vendors, and investors will quote.
The most visible shift is that GPU throttling is no longer a failure mode. It is the operating point. The panel described it as a deliberate design choice, supported by software and intelligent cluster middleware that watches power budgets and dials throughput accordingly. Vendors are selling the control plane for this. Hyperscalers are building it in-house. The argument is the same from both sides: a rack that runs predictably at 80% of nameplate power is more valuable than one that brownouts at 100%.
This reframes product launches. When a GPU vendor announces a new accelerator that draws 1,000 watts, the reader question is no longer "is that fast?" It is "what is the power-per-token ceiling inside a real cluster?"
The second shift is the slow return of the CPU. As AI workloads tilt from training to inference (that is, from teaching models to running them for users) more of the work is moving off GPUs and onto general-purpose processors, where memory bandwidth, not flops, is the binding constraint. Inference often runs on smaller batches and is more latency-sensitive than throughput-sensitive, which plays to the CPU's strengths.
Recent research on CPU-based inference and CPU-side serving systems like PowerInfer describe deployments where the right answer is not "more GPU." It is "fewer GPU, more CPU, and a memory subsystem designed for tokens in flight." The CPU was supposed to be a passenger in the AI era. It is regaining a steering wheel.
The third shift is the most technical, and it lives in the place where the AI model keeps its short-term memory. That memory is called the KV cache, and it is what lets a model continue a conversation, recall what it just said, and stay coherent across thousands of tokens of context. KV stands for key-value, the internal indexes a transformer uses to look up its own prior state.
KV cache is large. For a long context, it can be many gigabytes per request. GPUs have fast HBM memory for it, but HBM is scarce and expensive. CPUs have larger, slower memory attached. The industry is now building systems that park the KV cache in CPU-side memory when an AI agent is idle or between calls, and push it back to accelerator memory when the agent activates and starts generating tokens. This is most visible in agentic AI workflows, where a model sits waiting for a tool result or human input, and in mixture-of-experts models, where only some of a network's parameters run for any given request.
The vendors are not just talking about it. The research literature is filling in behind them: recent work on GDEV-AI explores heterogeneous-memory placement for mixture-of-experts models, while CPU-based deployment guides and CPU-side serving systems describe how to make that placement pay off in production. The point of the work is consistent: memory placement, not raw flops, is now a primary lever for power-per-token.
These three shifts interlock. Throttled GPUs raise the value of CPU-side work. CPU-side inference raises the value of memory placement. Memory placement, done well, lets an agentic system keep many idle sessions warm without burning accelerator power on them. Each of them is a workaround for the same underlying constraint: the building can only deliver so many watts, and the model is now expected to do more with them.
The practical lens for a reader outside the chip beat: when NVIDIA, Intel, Arm, AMD, or any hyperscaler announces an AI-infrastructure product in 2026, ask three questions. What is the rated power per token at the rack level? How much of the workload is GPU versus CPU? Where does the KV cache live when the model is idle? The answers will tell you whether the announcement is moving the industry toward more compute, or toward better compute per watt.
The chip industry spent 20 years racing for more transistors. The next race is for fewer watts per useful answer. The metric already has a name. Customers are already buying against it.