Robot Brittleness Is a Measurement Problem, Not a Data Problem

For years, the standard instinct in imitation learning has been that more demonstrations fix a brittle policy. The instinct is wrong, and the fix is to measure the gap between the right action and what the policy actually predicts under visual noise it should ignore.

The CSIRO preprint names that gap action drift: the change in a trained policy's predicted action when only the visual context shifts, even though the correct action should not. Lighting changes, a new background, a distracting object on the table: each should be invisible to the right behavior, but most visuomotor policies wobble anyway. D'urso et al. use that wobble as a sensitivity signal to rank candidate repair demonstrations offline, with no rollout success labels and no online execution required.

In MuJoCo bimanual cube transfer and SimplerEnv cube stacking, choosing 20 to 30 demos by this signal substantially outperformed matched random selection and approached the performance of much larger random repair budgets. The lesson travels. The same diagnostic frame applies to any policy that breaks on nuisances it should ignore: stop counting demonstrations and start measuring fragility. The brittleness problem ends when the data is chosen for the failure mode it actually addresses, not when the dataset gets bigger.