Why AI Data Centers Are Splitting Their Work Across Many Chips

When you send a prompt to an AI model tonight, the answer will be assembled by software shuttling work between several different kinds of chips. The speed, cost, and quality of the response depend on how well that handoff works, not on which chip is fastest.

The "fastest chip" framing is outdated. The binding question is whether the cluster can be stitched together efficiently, and that decides which AI products are economically viable. The shift is already visible in the data centers run by the so-called neoclouds, including DigitalOcean, CoreWeave, Lambda, and Verda, and in how hyperscalers are reorganizing their inference stacks around the limits of any one accelerator. The Future Of AI Compute Won't Run On Just One Kind Of Chip

Six engineering executives laid out the architecture in a Semiconductor Engineering roundtable: Satadal Bhattacharjee at Arm, Ashish Darbari at Axiomise, Moshiko Emmer at Cadence, Sharad Chole at Expedera, Cameron Brunner at Siemens EDA, and Sumit Vishwakarma at Synopsys. The thesis they share: AI compute is moving from homogeneous GPU-only clusters to heterogeneous clusters mixing CPUs, GPUs, NPUs (neural processing units, the dedicated AI accelerators that now ship alongside CPUs in laptops and phones), custom accelerators, and dedicated inference silicon, all stitched together by interconnect.

The structural change has a name: inference disaggregation. Instead of one pool of identical chips handling every stage of generating a response, a modern inference deployment uses separate clusters for each phase. A prefill cluster processes the incoming prompt. A decode cluster generates the answer one token at a time. And, increasingly, an agent-execution cluster runs the tool calls and intermediate steps an agentic request needs. Each cluster has different hardware and different software, and they are often connected by Ethernet.

Bhattacharjee, at Arm, described a five-layer architecture DigitalOcean has published that is tuned for heterogeneous clusters drawing on both AMD and Nvidia silicon, and pointed to Nvidia announcing a Groq 3 LPU (a language processing unit, a chip optimized for token-by-token generation) being used in a prefill cluster as evidence that even Nvidia is acknowledging that more than a GPU is required for some inference stages. That characterization comes from Bhattacharjee in the roundtable; Nvidia hasn't separately confirmed the specific prefill configuration.

The orchestrating software is where most of the new work sits. The base is the standard container layer that has been the default platform for years. On top of that, runtime, orchestration, scheduling, queuing, and serving layers decide which chip handles which part of a request. Hyperscalers use virtual cluster placement groups, a software construct that pins a set of workloads to a defined pool of machines, to keep a workload's data close to the chips that need it. HPE's Slingshot fabric, the InfiniBand interconnect standard in many GPU clusters, and 100 Gbps Ethernet are the wires beneath this software. Quality-of-service provisioning per workload, a way of guaranteeing bandwidth and latency to specific jobs, decides whether a real-time request gets served before a batch job.

The parallelization paradigms now in play each have different network topology needs. Tensor parallel splits a single matrix operation across chips, which needs the tightest, lowest-latency interconnect because every step depends on the neighbors. Data parallel runs the same model on different shards of a dataset, which can tolerate slower links. Context parallel, a newer approach, splits a long prompt across machines so each handles part of the input. Pipeline parallel assigns different layers of a model to different machines and feeds the output of one into the next. No single network handles all four well, which is why the wire question of what connects what is now as load-bearing as the chip question.

The memory wall is the second binding constraint. Chole, at Expedera, framed HBM (high-bandwidth memory, the stacked DRAM attached directly to a GPU die) cost and utilization as the critical bottleneck in modern GPU clusters. Long context windows push dynamic memory demand up faster than HBM capacity grows. The longer the context, the more memory a request holds open, the more the cluster's effective throughput drops. The economics of "context length" as a marketed feature run straight into that bottleneck.

Bhattacharjee drew an Apple-ecosystem analogy for Nvidia's position. The hardware is strong, but the moat is CUDA, the drivers, and the libraries that make the stack portable across generations. Heterogeneous compute doesn't break that moat; it forces every alternative to ship an equivalent software story. The neoclouds and software-layer startups, including Gimlet Labs and Together AI, are competing on orchestration, not on silicon, because the orchestration is where the heterogeneity is hidden from the developer and the end user.

What none of the roundtable speakers claim is that the end-to-end flow is solved. Co-packaged optics, where fiber connections move onto the same package as the switch chip, are emerging to address the heat and bandwidth limits of pluggable optical modules, but the standards are still settling. Brunner at Siemens EDA raised quantum machines as a future member of the heterogeneous cluster, which doubles down on the orchestration problem rather than relieving it. Power delivery and thermal envelopes at the rack level were named as constraints the software can't engineer around.

Token cost is the metric the operators are watching, because it sets the floor on which AI products can be sold at a margin. The DigitalOcean five-layer architecture is the first published template; the test over the next year is whether the rest of the industry converges on a similar shape, or whether each neocloud rebuilds its own.