Carnot compiles deep-research queries into inspectable execution graphs users can edit mid-run

Carnot: Interpretable, Interactive, and Optimized Execution of Deep Research Queries

Matthew Russo, Yash Agarwal, Tianyu Li, Zhuohan Gu, Michael Cafarella, Omar Khattab, Tim Kraska, Samuel Madden

cs.DB, cs.AI

2026-08-10

Carnot compiles NL analytics into editable notebook cells over a visible plan; editing one re-runs only it, and the optimizer picks implementations to meet cost and latency budgets.

What problem this solves

Enterprise analysts want to query data lakes in natural language, and neither of the two existing paths works well. Deep research agents take a question and return an answer with the entire chain of retrievals, transformations, and LLM calls hidden inside a black box; users can neither see where the run went wrong nor stop it from elaborating on a false premise. Semantic operators are transparent but bill per LLM call, which becomes unaffordable at enterprise data-lake scale. Analysts end up unable to verify intermediate results or steer the system, and any correction means rerunning the whole expensive pipeline.

Method

Carnot compiles a natural-language request into a physical execution graph and shows it to the user before execution starts. Each operator is a notebook cell viewable at three abstraction levels: natural-language intent, generated code or prompt, and raw input/output pairs. Execution is incremental and cached. Editing one cell re-runs only that operator while upstream results are reused. This directly addresses the core pain point: you stop paying full price for every iteration.

Key design choices:

• The QueryPlanner is itself an LLM agent that uses few-shot examples to compose plans in Carnot's operator vocabulary. Before planning, it dispatches a data-discovery sub-agent to sample data and analyze schema, so planning is not blind.
• The QueryOptimizer follows a Cascades-style framework and uses another LLM agent to pick the final physical plan from an estimated Pareto frontier of cost, latency, and quality. Given a user budget, it swaps implementations dynamically, for example replacing a large model with a smaller one for a low-precision filter.
• Operators come in two families: semantic operators (LLM-driven map/filter/join) and agentic operators (data exploration, Python code generation and execution), all attached to a core Dataset abstraction and chainable.

Results

This has to be said plainly: Carnot is a demo paper with no conventional quantitative benchmark and no numeric comparison to other systems. It demonstrates the interactive workflow on two workloads, an anonymized set of 1,000 consumer-product reviews and the legal task of the KramaBench academic benchmark, across three scenarios: inspecting the plan DAG before execution, adding constraints through chat to trigger replanning, and editing a prompt in a cell to watch the downstream recompute.

Code is open at github.com/mitdbg/carnot.

Why it matters

For teams building enterprise AI analytics, Carnot makes concrete a third path between black-box agents and expensive semantic operators: it keeps the flexibility of LLM agents while exposing the process so people can intercept, edit, and budget it. The incremental-execution-with-caching point is the most practical one. The dominant cost in iterative data analysis is repeated reruns, and re-running only what changed captures most of that saving.

Limitations

The authors concede the system offers no guarantee on plan accuracy or correctness. They sketch a path to soft statistical guarantees (borrowing from Lotus, giving guarantees relative to an oracle) but note it is soft because the oracle itself can be wrong. The demo format also means no end-to-end numeric comparison with existing deep-research systems; the savings are argued mechanistically rather than measured. The authors list several open items: web search, general tool calling, and improving the notebook UI to better reflect the non-linear DAG structure of the plan.

Terms

• Deep research agent: an LLM system that iteratively retrieves, aggregates, and synthesizes a report around a question.
• Semantic operator: a data operation implemented with an LLM, such as classifying a review as positive or negative, a judgment standard SQL cannot express.
• Physical execution plan: the concrete operator sequence and arguments a query compiles to, analogous to a database query plan.
• Cascades optimizer: a classic database query-optimization framework that searches a space of equivalent plans for the cheapest one.
• Pareto frontier: the set of solutions where no objective (cost, latency, quality) can improve without worsening another.

Source

• arXiv 2608.09532v1
• PDF
• Hugging Face
• DOI 10.14778/3827998.3828086

What people are saying

• MIT's Carnot: An Interactive Engine for Verifiable Deep Research Queries _reachsumit

Related papers

• Strongly universal string hashing at 0.2 cycles per byte beats Rabin-Karp and SAX
• Tsinghua reboots ACID transactions for LLM agents; a 0.6B confidence model adds 10.6 points over Claude Code
• RAG-Stack: Co-Optimizing RAG Quality and Serving Performance Across a Huge Config Space
• TEngineDB-V makes large-k vector search a native OLAP operator, up to 145x faster than StarRocks

All paper explainers