An Open Test Bed Asks Whether Training on Interventions Beats Training on Observations

Healthcare, policy, and climate questions share a stubborn shape: what would happen if we changed something, and what would have happened if we had not. A standard predictive model can forecast next quarter's hospital admissions, next year's emissions, or next month's case counts, but those answers stay inside the world that already happened. To act on the world, a model has to reason about worlds that did not.

A new open benchmark called DoTime, posted to arXiv on 2026-07-29, makes that gap testable. The release ships as a Python package named dotime on PyPI under an Apache 2.0 license, plus three frozen evaluation suites hosted on Hugging Face and Zenodo, and a 100,000-trajectory training snapshot scaled for foundation-model pretraining (arXiv 2607.27263; package metadata). The authors, Dennis Thumm and Ying Chen, frame it as the first benchmark built specifically for interventional and counterfactual time-series reasoning, the slice of causal-inference research they argue has been underserved by existing suites.

Two terms do most of the work in the paper, and both deserve a doorway. "Interventional" means training and testing on data where something was deliberately changed, the way a doctor would compare patients who got a drug to patients who did not. "Counterfactual" means asking the harder follow-up: given one patient who got the drug, what would their outcome have been without it. Standard machine learning usually sees only the first group; counterfactual reasoning requires pairing the two in a way that controls for everything else.

DoTime's mechanism is the structural inductive bias baked into how the training data is generated. Each simulated dataset comes from a known structural causal model, a recipe that says which variables cause which and how strongly. The package then injects interventions, including deterministic ramps, sinusoidal pulses, and regime switches, into the time series at known timestamps, and pairs the resulting trajectories with their un-intervened counterfactual twins. The test bed covers eight classical identifiability structures, the rules that say when a causal effect can be recovered from data, including back-door adjustment, front-door adjustment, and instrumental variables, all with exact ground truth attached (full paper).

The headline result is a single falsifiable claim: an intervention-trained prior-fitted network, a small neural network trained directly on simulated interventional and counterfactual data, beats an observationally trained prior-fitted network of identical capacity on the structure-matched held-out suites. According to the authors, that gap is positive in every identification structure, every trajectory length, and every seed they tested. Read carefully, the result is narrow. The comparison is structure-matched, so both models see the same causal graph at evaluation time, and the win is about inductive bias from the training distribution, not about discovering the graph from scratch. And "every seed" means three seeds per arm, not a swept hyperparameter study.

That limit is also the reason the three frozen suites matter. Each one is a fixed, downloadable dataset with a Zenodo DOI: dot-Identifiability-v1 with 10,800 episodes (Zenodo 10.5281/zenodo.20846064), dot-RegimeSwitch-v1 with 9,999 episodes (Zenodo 10.5281/zenodo.20846074), and dot-Continuous-v1 with 9,999 episodes. Anyone with a pip install can rerun the reported numbers, swap in a different model class, or try a new sampling strategy against the same held-out distribution. The "causal foundation model" language is aspirational; the package ships the test bed, not the model.

The use cases the abstract names are familiar: healthcare, policy evaluation, and climate science, all domains where the question on the table is whether a proposed change will help, and where the cost of guessing wrong is patient outcomes, public budgets, or climate stability. DoTime is not a system that answers those questions. It is a system for testing other systems that try to, with the data and the ground truth available to anyone who wants to check the answer.

The paper is a v1 preprint, hosted entirely on author-controlled infrastructure on arXiv, PyPI, Hugging Face, and Zenodo, and no independent replication is in evidence yet. The next pressure-checkable milestone is straightforward: a third-party run on the frozen suites with more seeds, additional model classes, and at least one structure-mismatched evaluation. The package is already pip-installable; the rest is a question of who shows up to run it.