Shift & Drift, an arXiv benchmark, tests the AI that steers self driving cars, finding sharp drop offs in imitation learning planners in new cities and under noisy controls, while a reinforcement learning baseline degrades more gracefully.
A new benchmark for self-driving motion planners exposes a failure mode that public leaderboards have not measured: the systems that score highest on familiar roads break when conditions shift. The benchmark, Shift & Drift, runs planners through two stress tests the field has not standardized before, and the preprint's findings point to imitation-learning systems degrading sharply, while a reinforcement-learning-based planner holds up better.
Shift & Drift is the work of an arXiv research team that built a two-track evaluation aimed at one problem. Today's top planners are usually trained and tested on data from a small number of cities, so it is unclear how they behave in a new urban layout, denser pedestrian traffic, or under compounding steering errors. The benchmark is the team's attempt to put numbers on that gap.
The first track, the Semantic Shift Track, converts the aerial DeepScenario Open 3D dataset, originally captured in Germany and the United States, into the nuPlan simulation framework. The result is 1,182 zero-shot scenarios spread across four German cities and San Francisco, heavy on pedestrian and cyclist interactions, that planners trained on North American and Singaporean data have never seen.
The second track, the State-Distribution Drift Track, leaves the cities alone and injects stochastic perturbations into the ego vehicle's dynamics. The point is to measure what happens when small actuation errors compound over time, the kind of drift that is invisible in a clean simulation loop but routine in production hardware.
Across both tracks, the paper reports a consistent pattern. Imitation-learning planners, the training paradigm that produces the highest scores on in-distribution benchmarks, show large performance drops under semantic shift, especially in pedestrian-dense scenes, and accumulate persistent errors when actuation noise is temporally correlated. A reinforcement-learning-based planner in the same evaluation degrades more gracefully, holding higher safety and progress scores through both stress tests.
The authors frame the result as a trade-off between imitation fidelity and closed-loop resilience. Training a planner to mimic expert demonstrations in known conditions produces a system that performs well on the test set it was designed for, and brittle when the test set changes. Reinforcement-learning training, which optimizes for long-horizon task completion under feedback, appears to give up some of that in-distribution polish in exchange for less catastrophic failure modes elsewhere.
For the AV industry, the directional finding adds weight to a running concern that benchmark leaderboard scores are a misleading proxy for deployment readiness. The paper does not claim a winner. It introduces a community benchmark and reports the empirical pattern the authors observed, on the planners they evaluated, in the two conditions they defined.
The full paper text beyond the abstract and introduction was not available in the captured source material, so the specific planner identities, exact numerical drop-offs, and ablations from later sections have not been independently verified. The peer-review status of the work is not confirmed. Treat the headline finding as a directional signal, not a verdict on which training paradigm will win.
The next move is the paper's: releasing Shift & Drift, with the DeepScenario-to-nuPlan conversion pipeline, as a community benchmark so other teams can run their planners through the same two stress tests. If imitation-learning systems keep failing in new cities and under drift while reinforcement-learning systems hold up, the gap becomes measurable. If they do not, that is a publishable answer too.