Training a large model is an exercise in managed catastrophe. Hardware fails. Networks stall. Preemption events interrupt jobs that have been running for days. The only thing standing between a team and starting over from scratch is the checkpoint — a saved snapshot of the model's state at a given point. Getting checkpointing right is the difference between a training run that recovers gracefully and one that wastes weeks of compute. Google published a technical explainer on March 31, 2026 detailing how continuous checkpointing in its Orbax library and MaxText training framework addresses the problem, available at Google Developers Blog. The explainer is not marketing material. It is an honest account of a hard ops problem that every organization running large-scale training has solved, or suffered, in its own way.
The conventional approach is fixed-interval checkpointing: save every X steps or every Y minutes. The explainer identifies the failure mode concisely: set the interval too high and a hardware failure wipes out hours or days of work. Set it too low and the checkpoint process itself becomes a bottleneck — blocking training while the save completes, especially on unstable networks. The team that wrote this has apparently run into both problems enough times to want to document the solution.
Continuous checkpointing works differently. Rather than checkpointing on a fixed schedule, Orbax initiates an asynchronous save only after the previous save operation has completed successfully. This means checkpoints are generated as frequently as the system can sustain without blocking training — maximizing I/O utilization while minimizing the risk that a failure wipes out recent progress. The key insight is that the optimal checkpoint frequency is not a fixed number; it is whatever the system's I/O can sustain while the job is running.
The benchmark in the explainer uses two slices of v5p-128 — 64 chips per slice — running the Llama 3.1 70B model. Continuous checkpointing produced markedly smaller P50 checkpoint intervals compared to checkpointing every 100 steps, with an expected increase in average training step time due to more frequent device-to-host data transfers. The efficiency gains are more pronounced at larger scale. Large model files are segmented into smaller fragments for distributed training, which reduces device-to-host blocking time per fragment. And mean-time-between-failure scales inversely with cluster size — larger training runs fail more often, which means the cost of infrequent checkpointing grows with the scale of the job.
The engineering insight worth dwelling on: continuous checkpointing does not eliminate the trade-off between reliability and performance. It reframes it. Instead of choosing a fixed interval and accepting either risk or overhead, teams configure the system to be as aggressive as the I/O allows, and let the completion-triggered logic handle the rest. Orbax's implementation also allows customizable preservation policies — teams can keep checkpoints based on evaluation results, not just recency, which matters for research workflows where a mid-run evaluation might identify a configuration worth preserving even if it is not the most recent state.
This is ops infrastructure journalism. The kind that does not make headlines but that every team running large training jobs has a war story about. The Orbax explainer is notable precisely because it is written from the inside of the problem rather than the outside looking in. Whether Google's approach generalizes to other frameworks — PyTorch FSDP, Megatron-LM — is a question the explainer does not answer. The principles translate; the implementation details are Orbax-specific.
The explainer is worth reading in full for anyone operating at this scale. It is also a reminder that the headline capability of a model — what it can do when trained — depends on infrastructure that nobody talks about: the checkpoint manager that decides whether a hardware failure is an inconvenience or a catastrophe.
