AI Infrastructure Is Moving Past Bigger GPUs. The New Fight Is Memory and Networking

Alibaba's 10,000-card compute cluster was framed in a recent Semiconductor Engineering round-table as a workaround for limited access to top-tier Nvidia parts. Read it as a workaround in two directions: scaling around the chip shortage, and scaling around the chip itself. Once a cluster is treated as a system rather than a count, the design problem changes. It becomes a question of how memory sits next to compute, how fast data can move between accelerators, and how the orchestrator schedules work across silicon that isn't identical.

The panel, which drew architects from Arm, Axiomise, Cadence, Expedera, Siemens EDA, and Synopsys, converged on a single thesis. The binding constraint inside an AI data center has moved from raw FLOPs to the fabric that ties chips together, the high-bandwidth memory (HBM) stacked on top of them, and the orchestration layer that decides which chip handles which stage of a request.

That shift is why heterogeneous computing, the practice of mixing different processor types inside one workload, is becoming the default. Inference isn't one job; it is a pipeline of stages. The prefill stage, where a model ingests a prompt, is compute-heavy. The decode stage, where a model generates tokens, is memory-bandwidth-heavy. Tool execution, increasingly common as agentic workflows, can fall back to CPUs. Putting all of that on a single GPU class wastes the parts of the silicon that aren't doing useful work. The new design grammar mixes GPUs, custom ASICs, neural processing units (NPUs), and language processing units (LPUs) targeted to specific stages, then routes traffic between them.

Memory is where the cost and the constraint now sit. HBM, the stacked DRAM that lives next to an accelerator and feeds it data, is in tight supply and dominates accelerator bill of materials. Several panelists pointed to memory bandwidth, not peak FLOPs, as the wall scaling runs into. When the bottleneck is feeding the chip rather than running instructions, the answer isn't a faster GPU. It is a better placement of memory and a tighter network around it.

Interconnects are the third leg of the stool. Inside a server, accelerators talk to each other and to the host CPU over NVLink-style links. Between servers and racks, UALink, an emerging open accelerator-to-accelerator standard, is the main challenger to Nvidia's proprietary NVLink and to Ethernet-based alternatives. Nvidia's response is NVLink Fusion, which opens parts of NVLink to outside accelerators. The two standards are not the same, and both are still landing; UALink adoption is projected, not confirmed as a default.

Hyperscalers are not waiting. Google runs its TPU fleet on a custom optical interconnect called ICI. Amazon has built Trainium. Meta has MTIA. Microsoft has Maia. Each system comes with its own fabric, its own orchestration, and its own memory topology, and each one is the hyperscaler's escape route from depending on a single supplier's roadmap. NeoClouds, the smaller AI-focused cloud providers, are deploying multi-layer inference fabrics that mix AMD and Nvidia silicon under a single orchestration plane. The data center is becoming a federation of specialized pools rather than a single GPU grid.

The agentic turn makes orchestration harder, not easier. Tool-calling and parallelization strategies (tensor, data, context, and pipeline parallelism) all assume different network shapes, and MCP-style tool servers, the Model Context Protocol endpoints that hand models external tools and data, increasingly run on the CPU side of the rack. Picking a topology is no longer a hardware question; it is a workload question.

A useful falsification test: if a major hyperscaler returns to a Nvidia-only stack and cluster scale stops mattering, the thesis here is wrong. The bets being placed right now say that won't happen. HBM supply, UALink deployment, NVLink Fusion licensing, and the hyperscaler escape hatches are all moving in the same direction. The next eighteen months of AI infrastructure headlines will read less like chip launches and more like wiring diagrams.