What Happens When the Model Eats the Stack? Rethinking the Research Agenda for Data Agents to Withstand the Bitter Lesson
Liana Patel, Siddharth Jha, Negar Arabzadeh, Carlos Guestrin, Ion Stoica, Matei Zaharia
cs.DB
2026-09-03
On DAB, GPT-5.6 Sol coding agents beat o3 by 35+ points and cut turns ~4×. Purpose-built scaffolds recede; a 12-trace offline semantic context adds 19 points.
Sutton's bitter lesson is a systems warning: once a general learner can absorb a skill, the hand-built layer that used to supply that skill becomes dead weight. Database research has spent two years adding planners, SQL generators, and multi-agent workflows so data agents can survive a single hard query.
UC Berkeley and Stanford ask a different question. After the model swallows those layers, what systems work is still worth doing? They run o3, GPT-5, and GPT-5.6 Sol on TAG-Bench and DAB, pitting a general Codex coding agent against the best purpose-built data agents they can assemble. The claim is blunt. Task-level scaffolding is being internalized. What does not go away is knowledge of a particular data environment, and that knowledge gets relatively more expensive as the rest of the loop shrinks.
Two experiments.
Capability trend. TAG-Bench mixes exact computation, semantic reasoning, and world knowledge over relational databases. DAB scores multi-step analytics over fragmented enterprise sources. The generalist is always Codex, with no task-specific engineering. The specialist on TAG is an approximation of Agentar-Scale-SQL, a BIRD leaderboard system whose full inference stack is unpublished; they reuse the public SQL prompt plus one execution-guided revision, and only on 60 non-aggregation queries. On DAB, Agentar is not built for multi-turn work, so they run DeepEye, the next-best open BIRD system, with added DuckDB and MongoDB connectors. They sweep reasoning levels and report low reasoning.
Context that survives. GPT-5.6 Sol writes a persistent semantic context offline from 12 development traces, one per database. The artifact is frozen and prepended to the first user prompt on 42 held-out DAB tasks. Three self-authored targets: Accuracy, Latency, Schema. GEPA grows an accuracy-oriented instruction from the same 12 tasks. The control is No Context: the agent rediscovers the environment on every query.
With o3, the Agentar-style specialist still wins on TAG, both on accuracy and token cost. With GPT-5.6 Sol, the coding agent beats the same-model specialist on both benchmarks. On DAB the GPT-5.6 Sol coding agent sits more than 35 points above the o3 coding agent and is more than 2× more token-efficient. On TAG it sits next to the Oracle of expert queries running on LOTUS. Figure 1 is a scatter plot; the paper does not publish a numeric table.
Turns fall faster than accuracy rises. On DAB, mean turns go from 23.2 (o3) to 9.9 (GPT-5) to 6.0 (GPT-5.6 Sol), about 4×. The cuts land on analytical queries and on verification and recovery. Schema exploration's share of the remaining work grows from 16% to 25%. Failures move with that mix. Execution errors shrink. Of 40 GPT-5.6 Sol failures, 40% are semantic misreads (C1), 18% pick the wrong source or field (C2), and 38% botch entities or join keys (C3). The paper groups those as environmental knowledge and says they account for over 60% of remaining failures.
Offline context can amortize that cost, with a bill attached.
| Method | vs No Context | Mean turns | Build |
| Self-Curated (Accuracy) | +19 points | 6.0 | 164.9s / $1.09 / 2.52KB |
| Self-Curated (Latency) | below the Accuracy target | 4.5 | 154.9s / $1.13 / 2.25KB |
| Self-Curated (Schema) | modest accuracy drop | 4.6, schema share 25% → 9% | 3413.6s / $9.60 / 162KB |
| GEPA (Accuracy) | between No Context and Accuracy | 4.9 | 1359.8s / $12.16 / 7.55KB |
The Accuracy note lifts DAB by 19 points at almost the same turn count. The Latency note cuts turns to 4.5 and is still slower wall-clock than No Context. The Schema note is the one that actually stores environment knowledge; exploration collapses, accuracy falls. The authors blame overfitting 12 traces and then trusting the note too much.
If you ship internal data agents, stop adding planners to cover model weakness on a single query. Planning, code, tools, and debug are going into the weights. Metric definitions, canonical tables, join paths, and conflicting business vocabularies are not. Re-deriving them per query becomes the dominant waste.
The paper also pushes against the CIDR 2026 premise in Supporting Our AI Overlords: that data systems will drown in speculative agent SQL. Newer models issue fewer, better queries. The system problem that grows is a semantic layer that goes stale.
This is an agenda paper, not a new system. The only artifact you can copy tomorrow is "let the agent write a markdown file offline." The 19-point bump is on 42 held-out queries and 12 databases. Semantic consistency protocols, context data structures, and compression are sketched, not measured.
The specialist baselines are thin. Agentar's unpublished multi-generator, revisor, and selector are missing; aggregation queries are dropped; DeepEye is a next-best open system with extra connectors. Some of the early specialist lead may be an artifact of a stripped baseline.
Failure labels are assigned by GPT-5.6 Sol, the same model family under test. Context experiments use 12 traces and 12 datasets; the Schema variant already overfits. The authors themselves say real stores are terabytes and thousands of tables. Only OpenAI models and Codex appear. The Latency context uses fewer turns and still loses on wall-clock latency; the paper does not break that gap down.
Section 3 on consistency scope, incremental versus holistic refresh, physical design, and compression is a design space with no measurements.