Light model hits 31-39% on MIT's BEAVER text-to-SQL benchmark vs 11.4% in the paper
Temporary_Papaya1058 · reddit · 2026-11-02
BEAVER is MIT's text-to-SQL benchmark built from real enterprise data warehouses, where the paper's best setup scores 11.4% execution accuracy. A developer ran schemagate (open-source schema selection) with Gemini 2.5 Flash on 100-question samples and reached 31–39% (dw: 31%, second sample 35%; neutron: 39%), with 97.7% table recall.
What moved it:
- A bug fix: dw declares no primary keys, so schemagate inferred zero joins; accepting same-named key columns restored 0→55 joins.
- Bridge tables sharing no words with the question: a new expandinward option adds tables whose foreign keys point into the selection.
- Example retrieval: three similar solved questions from BEAVER's public split — gold SQL encodes warehouse conventions (term codes, status keys) not guessable from the schema.
Caveats, honestly stated: the paper's 11.4% uses no examples, so it's not apples-to-apples; excluding 6 exact-duplicate examples drops scores to 31.0/34.7/38.5; not an official leaderboard result (sent to the BEAVER team). Both fixes shipped in 1.15.4 (pip install -U schemagate).
More from coding & agent
- From chatbot to agent: how tools, planning and memory were added step by step — _jaydeepkarale · 2026-11-04
- Binance's $7B hack reportedly started with a tampered Homebrew install during vibe coding — xuanalogue · 2026-11-04
- $100 of GPU time edits model weights to cut thinking tokens by 40% — jarrodwatts · 2026-11-04
- Cua Driver integrates with Google Antigravity for background cross-app computer agents — m4rkmc · 2026-11-04
- Sentdex: OpenHands was the first reasonably useful agent to abstract away the code editor — rajistics · 2026-11-04
- Security advisory: run AI agents in managed cloud environments, not employee endpoints — amasad · 2026-11-04