The Benchmark Wasn't Broken. The Test Was.

Every shared testbed in machine learning eventually runs into the same wall. Call it the leaderboard noise problem: the moment researchers can finally compare methods on identical conditions, they discover that small, unstated modeling choices were doing more of the ranking work than the algorithms were. A standardized testbed does not settle a field; it exposes what was actually being measured.

The RideGym paper, an open simulator for ride-hailing dispatch, makes the wall visible. Its headline figure — a one-hour, city-scale simulation across thousands of vehicles and tens of thousands of orders running in under a minute — is real, but it is the easy half. The harder finding, buried in the authors' own validation, is that swapping a single knob (the exploration noise used during training) was enough to reshuffle the relative ranking of dispatch methods that had been calling themselves state of the art — one read of the data is that the field was measuring noise all along.

The mechanism generalizes. Build a shared harness, pin the conditions, swap one hyperparameter, and the leaderboard moves. Strongest read first: the field was measuring noise all along and calling it progress. Second read: the algorithms are roughly equivalent and the room to differentiate is mostly in the experimental setup. Either way, the testbed is the news — not because it picks a winner, but because it proves the prior contest was not measuring what it claimed to.