BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence
Chuxuan Hu, Yeye He, Penny Zhou, Wee Hyong Tok, Daniel Kang, Surajit Chaudhuri
cs.LG, cs.AI, cs.CL, cs.DB
2026-09-17
On 100 real Power BI questions, o4-mini hits 48.2% SQL without tools. Search, join and reshape tools lift GPT-OSS-120B from 5.3% to 45.3%; a post-trained 8B reaches 35%.
Real work in Power BI or Tableau is not "write SQL against a clean table." Analysts first hunt relevant tables in a pile of raw files, reshape them with unpivot or transpose, click-define joins, and only then drag out a chart. NL2SQL benchmarks such as BIRD and Spider skip the first three steps: tables are already washed and the schema is already aligned. Those three steps are where enterprise users actually burn time.
UIUC and Microsoft Research crawled more than 3,000 public .pbix projects and hand-extracted 100 (business question, exported result table) pairs from dashboard visuals, producing BI-Bench. A project averages 10.6 tables (max 52), about 80k rows (max over 7 million), and 12.75 joins (max 94). The question they pose: given a bag of raw tables and a natural-language question, can an LLM search, reshape, join, and analyze on its own, and emit the right result table.
BI-Agent wires existing data-management algorithms into an agent tool loop, so the model does not have to guess from a handful of sampled rows.
Post-training uses synthesized trajectories. From real BI projects held out from the test set, they sample a connected sub-schema, materialize a wide table, have an LLM invent a question and code against that wide table, and keep the executable result R. Relevant and irrelevant tables are then mixed, with occasional inverse reshapes, so the student has to search and transform. Training queries that sit too close to test queries are filtered (only 0.28% of training queries are nearer a test item than that item's nearest neighbor inside BI-Bench). SFT uses GPT-4o as teacher and keeps only traces whose execution matches R exactly. RL uses GRPO: +1 for a match, -0.5 for a wrong non-empty table, -1 for no table, and -0.1 per syntax-error iteration. Gradients are applied only to tool-call blocks. Qwen3-8B took about 9 hours of SFT and 22 hours of RL on 2x H100, under $200 to train.
Evaluation matches table shape (aggregation grain) and cell values, allowing row and column permutation, with numeric tolerance rtol=1e-2, atol=0.5. Each query is run 10 times; paired t-tests sit on the 100 tasks.
Twenty-four models and systems, in both SQL and Python.
| Model | SQL, no tools | SQL, with tools |
| o4-mini | 48.2% | 61.9% (+13.7) |
| GPT-5.5 | 46.7% | 60.2% (+13.5) |
| GPT-OSS-120B | 5.3% | 45.3% (+40.0) |
| Qwen3-8B | 5.2% | 19.8% (+14.6) |
| Qwen3-8B-RL-Tool | 5.2% base | 35.0% (+29.8 vs base) |
Tools add 14 points on SQL and 11 on Python on average, significant in 19 of 20 comparisons. Without tools, GPT-OSS-120B stops after a median of 3 model calls and barely inspects the data; with tools the median rises to 9, and those calls replace guesses with observations.
The post-trained 8B: SQL without tools reaches 22.6% after SFT and 23.5% after RL; with tools, 27.2% then 35.0%. The top four open-source NL2SQL models on the BIRD board score 6.0% to 17.3% SQL on BI-Bench. The two strongest open-source agents on Spider 2.0-lite (ktx with GPT-5.5, Databao with GPT-5.2) land at 26.3% and 23.8%.
On held-out Spider 2.0-lite (122 local queries), Qwen3-8B SQL goes from 0.8% to 16.4% with RL plus tools, about 20.5x. Error analysis: tools cut transform errors by 54.2% and table-selection errors by 40.9%. Missing information (dropped columns, rows, or fields) remains the leading failure.
Cost: Qwen3-8B-RL runs the whole BI-Bench for about $0.19; o4-mini and GPT-4o exceed $10, roughly a 54x gap. Tools also drop mean interaction rounds from 7.2 to 6.3.
The message for data teams is specific. NL2SQL agents that score 70 on warehouse-ready tables fall to the low 20s on raw .pbix sheets. The bottleneck is not writing SELECT. It is unpivoting a crosstab that came from Excel, and finding the right snowflake join among 30-plus tables. Wiring join and reshape predictors the database community already built is worth more than scaling the backbone one more notch. An 8B with tools and post-training hits 35% SQL, still short of o4-mini with tools at 61.9%, but inference is an order of magnitude cheaper. That is a fit for long-tail, one-off analysis with no semantic layer, not a replacement for a standing enterprise BI stack.
BI-Bench is 100 items from 82 projects, more than 400 person-hours of labeling. Questions are paraphrased from visualization titles, with implicit filters the authors patched in, which is still a distance from a business user thinking out loud. Scoring checks the result table, not how a dashboard should look. The authors define the problem as one-off Q&A over raw tables and say it is not meant to replace mature warehouse BI.
The tool set is incomplete: missing information is still the top error, so search and join still drop measures and filters. Ablating the RL reward drops Python-with-tools accuracy from the SFT 22.6% down to 19-21%, so the reward is tightly coupled to the result. Synthesized training questions may still sit in a different distribution from real dashboard prompts, and 16.4% on Spider 2.0 is not production-ready. Public .pbix files also skew toward tutorials and portfolios, not the dirtier internal models in a bank or a supply chain.