AIBRIEF
Researchers built a benchmark from real business dashboards. Frontier AI still scored below 50%, and the tool that closed some of the gap shows how far 'AI replaces the analyst' really is.
Frontier AI scored below 50% on a new benchmark built from real business dashboards, according to a preprint from researchers introducing BI-Bench. The paper argues the result is the most direct measure yet of how close AI is to doing a data analyst's end-to-end job.
End-to-end business intelligence means more than running a query. The traditional workflow, used in tools like Power BI and Tableau, requires a person to find the right tables, reshape the data, build the join relationships across them, and only then answer the business question. The paper frames that prep work as the tedious burden its system, BI-Agent, is designed to remove.
BI-Agent is a tool-augmented model that decomposes the workflow into subtasks, from search to join to transform, and orchestrates specialized data-handling methods. The authors report that giving a vanilla large language model these tools raises accuracy by up to 40 percentage points. A second pass of post-training on real BI trajectories, using both supervised fine-tuning and reinforcement learning, adds up to 30 more.
Even with both steps stacked, the system does not clear 50%. The paper points to the prep work as the part that still has to be done by hand. The benchmark and code are public on GitHub. The paper is a preprint and has not been peer-reviewed.