AINEWS
Speech to text can score well on average and still fail particular speakers. A new 4,888 speaker Hindi test set is the first major benchmark built to expose that gap at the scoreboard level.
Commercial speech recognition has been measured roughly twice as bad for Black speakers as for white speakers, with further gaps by gender, age, and accent. That finding, from a 2020 Stanford study of five major systems, has been replicated and extended in the years since, including work that quantifies how automatic speech-to-text performance varies across those same axes (Hugging Face blog: The Open ASR Leaderboard Adds Its First Global South Language).
The Open ASR Leaderboard, a Hugging Face community project that ranks open-weight speech-to-text models, has now added a benchmark designed to make those gaps visible at the scoreboard level. The new Monsoon evaluation sets, available in Indian English (en-IN) and Hindi (hi-IN), are the first Indic-language tests on a multilingual tab that previously covered only European languages. They are also, more importantly, the first major speech-to-text benchmark built around a specific principle: a single average accuracy score can be right on average and wrong for a particular population.
Hindi, spoken by more than half a billion people, is the headline addition. The structural innovation is the test design. The four Monsoon splits together cover 4,888 speakers, with 12 attributes recorded per speaker. The collection was built to vary along nine axes: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and the existence of multiple valid transcripts for the same audio. Each set is released as a public split that anyone can self-score against, plus a private split that is withheld to limit benchmark-specific overfitting.
Word Error Rate (WER), the standard accuracy score for speech-to-text, has long been treated as a single number. That treatment is the problem the new benchmark is built against. Prior test sets on the Leaderboard, like the broader literature, recorded what was said, not who said it. The Monsoon sets record both, which is what makes the 12 attributes load-bearing rather than decorative. A model that scores well on the public split can now be checked against age, region, device, and accent separately. The Leaderboard has, in parallel, added held-out private splits, benchmark-fitting analysis, and what its operators call gap-closing in normalisers, all moves designed to make the headline number harder to game.
The risk note is in the source itself. The Monsoon design still collapses performance to one number per speaker group. It is, in other words, a disaggregation move, not a distribution move. The Leaderboard will tell you whether a model fails more for older speakers or for certain districts, but it will not, yet, publish a per-speaker error distribution. The wider problem the benchmark is built to surface, that evaluation infrastructure has historically been a poor fit for the populations it claims to measure, remains partly intact.
For developers, the practical change is straightforward. Any model submitted to the Leaderboard can now be scored on Monsoon en-IN and Monsoon hi-IN, with separate public and held-out results. The held-out split is the one that matters for bragging rights, since it is the operator's number rather than the submitter's. For the rest of the field, the open question is whether comparable disaggregation will be added to the existing European-language sets, or whether Monsoon will remain a one-off demonstration that aggregate WER is a category error when subgroups differ.