SIGNAL
The execution-accuracy number that AI SQL vendors put on their slides is a known-bad proxy for whether the system is actually right. A query that runs and returns rows is not a query that returned the right rows, and that distinction is doing all the work in the new enterprise benchmark ESQ-Bench.
Mishra, Chukkapalli, and Naik's 12 June 2026 paper tested GPT-4o and Claude Sonnet 4.6 on three tiers of enterprise schemas, ranging from 95 to 227 gold-validated question-query pairs across hundreds of tables and four SQL dialects. The headline figure: 73 to 99 percent of queries the top models successfully executed on the harder enterprise tiers still returned the wrong answer. That is the silent-divergence rate, and it is what the standard leaderboards never measure.
Academic benchmarks use small, clean schemas and judge only whether a query runs. Real enterprise databases have hundreds of tables, mixed SQL dialects, and questions that demand cross-schema joins. The model executes; the rows are syntactically valid; the join targets the wrong table or applies the wrong filter; the user gets a confident wrong number. Spider and BIRD never see this failure mode because their schemas make it nearly impossible.
The closing stake: any organization letting an AI assistant touch production data should ask vendors for results on enterprise-schema benchmarks and on silent-divergence rates, not for execution accuracy on academic tests. The wrong KPI is now a known-bad KPI.