Distributed training teams have been guessing at checkpoint frequency for years. Google's Orbax and MaxText teams have a suggestion: stop guessing.
The companies introduced continuous checkpointing on March 31, 2026, converting what was a manually tuned checkpoint_period parameter into a background drain. When enabled, Orbax starts an asynchronous checkpoint save only after the previous save finishes, rather than waiting for a timer to fire. No more second-guessing whether 100 steps is too aggressive or 1,000 is courting data loss.
The approach handles the two failure modes that checkpointing has always involved. Run too infrequently and a node failure wipes out hours of work. Run too aggressively and synchronous saves block training entirely — the network can't keep up with the data moving from accelerator to host to storage. Continuous checkpointing tries to thread that needle by always saving, but only saving one at a time.
Multi-slice training gets a specific optimization. Orbax confines checkpointing to slice 0 and the storage server, leaving inter-slice communication unblocked. On a two-slice v5p-128 cluster running a llama-3.1-70B continuous pre-training task, this means slices 1 through N keep talking to each other while slice 0 handles the storage I/O. The blog describes this as keeping "inter-slice communication unblocked and unaffected."
One explicit constraint: the storage bucket needs to be co-located with the training cluster. Google's documentation is unusually direct about this. "Utilizing a cross-metro network can significantly degrade checkpointing speed," the blog reads. That is not a recommendation — it is a hard requirement baked into the architecture.
The blog also acknowledges an expected cost: average training step time increases because the system transfers data from device to host more frequently. The efficiency gains at large scale come from smaller model file fragments reducing blocking time, but the step time overhead is real and not fully quantified in the post. Google does not provide a concrete P50 interval or a step time overhead percentage. That is a gap the blog itself does not paper over.
Google notes that mean time between failures scales inversely with the magnitude of the scaling operation — double the chips, halve the MTBF. At the cluster sizes required for frontier training, failures are not exceptional events. They are expected operational reality. Continuous checkpointing is the answer to that reality: not prevent failures, but make them cheap.
The policy layer lets teams define what to preserve. Orbax provides an EveryNSeconds policy as an example, with a 180-second interval, and the abstract policy interface allows custom logic tied to evaluation results. Teams can decide what checkpoints are actually worth keeping rather than hoarding everything.
There is context the blog post does not provide. PyTorch shipped asynchronous checkpointing in June 2024, demonstrating the pattern on a 7B parameter model where checkpointing downtime dropped from 148 seconds to 6.3 seconds. Google was not first to this idea — the company was roughly twenty months behind the open-source ecosystem in shipping an equivalent pattern for its own training stack. The Google Developers Blog post does not acknowledge PyTorch's prior work.
What Google does offer is scale: the benchmark runs on 64 chips per slice, and the blog describes efficiency gains as "amplified at large scale." The fragment-based approach — segmenting model files into smaller pieces to reduce device-to-host blocking time — is the mechanism. That is a real optimization, and it is specific to configurations where the model file itself is large enough to fragment meaningfully.
Continuous checkpointing overrides the checkpoint_period knob entirely. The old parameter is still there in the config, but enabling enable_continuous_checkpointing: True ignores it. For teams that have spent cycles tuning that number against their specific network topology and workload, that is a quiet breaking change worth knowing about before upgrading.
The feature is real infrastructure solving a real operational problem. Whether Google's implementation outperforms the equivalent open-source patterns that have been available since mid-2024 is a question the blog post does not answer — and that silence is itself informative.
