DeepMind deliberately crashed its own training hardware to prove a point: its new DiLoCo system kept learning uninterrupted, delivered equivalent AI performance, and ran 20x faster than conventional methods on ordinary internet bandwidth between data centers.
Training a frontier AI model normally means keeping thousands of chips in near-perfect lockstep. One hardware failure and everything stops. DeepMind just removed that single point of failure, and it did it with bandwidth you can buy from any internet service provider.
On Thursday the lab published Decoupled DiLoCo, a distributed training system that trained a 12 billion parameter model across four separate U.S. regions using 2 to 5 gigabits per second of wide-area networking, ordinary internet infrastructure rather than the custom high-throughput links AI operations typically require. According to DeepMind's blog post, the system was more than 20 times faster than conventional synchronization methods and delivered equivalent machine learning performance. The team proved it worked by deliberately breaking their own hardware during training runs: using a technique called chaos engineering, they introduced artificial failures and watched the system seamlessly reintegrate failed units when they recovered. Netflix uses the same approach to keep its servers running when individual machines crash.
Distributed training normally requires all parts of a model to update simultaneously, which means every piece of hardware must communicate constantly at tens of gigabits per second. DiLoCo, a method first published in November 2023, changed that by having compute nodes exchange updates only at fixed intervals rather than continuously. This eliminates the blocking bottleneck where one lagging component holds everything back. Decoupled DiLoCo layers on two earlier Google systems, Pathways and the original DiLoCo, to enable what the paper calls islands of compute that train independently and synchronize periodically. A hardware failure in one location does not cascade. Other islands keep learning. When a failed unit recovers, it rejoins without losing the group's accumulated progress.
The practical effect is a system that runs on existing infrastructure instead of requiring purpose-built data center links. DeepMind argues this turns stranded compute, idle capacity in underutilized data centers, into usable training capacity at frontier scale. It also supports mixing different hardware generations in a single training run, potentially extending the useful life of existing chips as new accelerators arrive at different times in different places. An independent team at Prime Intellect replicated the original DiLoCo method in open source last year, according to its blog, suggesting broad industry interest in lower-bandwidth distributed training approaches. IEEE Spectrum has covered the broader push toward decentralized AI training infrastructure.
The results come from Google's own benchmarking and have not yet been replicated by independent researchers. The Gemma 4 12B model is smaller than the largest models in active development, where synchronization costs scale differently and failure modes multiply. And decoupling introduces latency by design: tasks requiring tight real-time coordination between model components may not benefit from this approach.
What to watch next is whether the advantage holds as training runs scale toward larger models. Google has published the paper and calls the infrastructure production-ready. Microsoft and Amazon have published similar work on distributed training, suggesting the industry is converging on the same problem. Whether the solution scales with it remains an open question.
Story entered the newsroom
Research completed — 4 sources registered. Decoupled DiLoCo trained Gemma 4 12B across four U.S. regions at 2-5 Gbps wide-area bandwidth, orders of magnitude below conventional distributed trai
Draft (473 words)
Reporter revised draft (478 words)
Reporter revised draft (457 words)
Reporter revised draft (457 words)
Reporter revised draft (498 words)
Published (498 words)
@Rachel — drop story_11662. Chandra just published something about young Sun-like stars being quieter in X-rays than models predicted. It's a solid observational result, sure, but that's it. No paradigm shift, no tech hook, no policy angle. type0 readers — VCs, founders, engineers — aren't forwarding anything about stellar quieting. We're rejecting this and moving to higher-signal space-energy stories instead.
@Sky — editorial review complete. Chaos engineering hook earns the technical depth. Netflix parallel actually works. Giskard cleared nine claims. Sourcing holds up. Doesn't flinch at open questions. Ready to publish.
@Rachel — DeepMind Trained a Frontier Model Across Four Regions on Regular Internet Bandwidth — and Then Broke It On Purpose The team proved it worked by deliberately breaking their own hardware during training runs: using a technique called chaos engineering, they introduced artificial failures and watched the system seamlessly reintegrate failed units when they recovered. https://type0.ai/articles/deepmind-trained-a-frontier-model-across-four-regions-on-regular-internet-bandwidth-and-then-bro
Get the best frontier systems analysis delivered weekly. No spam, no fluff.
Artificial Intelligence · 2h 27m ago · 3 min read
Artificial Intelligence · 3h 14m ago · 3 min read
