Apodex Releases TRACES: A Benchmark for AI Scientific Discovery on Open Questions

On September 4, Apodex (founded by Tianqiao Chen) released TRACES, a new benchmark positioning itself under a "Discoverative AI" paradigm. Endorsed by researchers including dair-ai and Omar Sanseviero (@omarsar0), it is the first systematic attempt to evaluate research agents making credible discoveries on open scientific questions whose answers are not yet confirmed.

Confirmed

• TRACES does not test whether models answer known questions; it measures whether they can rigorously and evidence-based advance unknown problems. Its name is an acronym of six capabilities: the first three assess the exploration process, the last three assess conclusion quality—Coherence (consistency across long investigations), Evidence (every claim traceable to real sources), and Scope (the solver states the boundaries of its conclusions).
• The 423 problems were hand-crafted by 10 PhDs, characterized by "the correct answer may not yet exist."
• Evaluation uses dual verifiers: an outcome verifier checks only results against hidden ground truth, while a process verifier reviews logged operations, errors, and corrections without seeing result scores.
• Per @SucceededMind, TRACES claims "the era of static benchmarks is over": it evaluates the agent's full execution loop—tool selection and strict tool execution, dynamic error repair, and long-context handling—rather than comparing final outputs to hidden answers.
• Process scores also drive a repair loop: per Apodex's report relayed by omarsar0, flagged runs generate repair notes, and solvers retry without seeing answers; re-running failed trajectories yields an average gain of 0.155. In an AAV capsid design case, the process-verifier-driven repair loop surpassed SOTA.
• The benchmark targets a blind spot: HLE, FrontierMath, MMLU, and BrowseComp are hard but assume answers exist; models that ace them can still stall on genuinely open research.

Why it matters

• TRACES fills the scientific-discovery dimension traditional benchmarks cannot measure, shifting evaluation from "answering known questions" to "rigor in exploring the unknown."
• The process-verifier-driven repair loop demonstrates that process scores can do more than grade, potentially shaping the training and evaluation of next-generation research AI.

2026-09-04 ~ 2026-09-05 · 9 related posts

• Apodex launches TRACES, a benchmark measuring whether AI can rigorously explore the unknown — eyishazyer · 2026-09-04
• [source] Apodex's TRACES benchmark: 10 PhDs hand-built 423 open research problems — omarsar0 · 2026-09-04
• TRACES' last three axes grade conclusions: coherence, evidence, scope — omarsar0 · 2026-09-04
• Apodex reports process-verifier repair loop beats SOTA in AAV capsid design (+0.155 over 434 retried runs) — omarsar0 · 2026-09-04
• [source] Process scores drive a repair loop: failed trajectories gain 0.155 on rerun — omarsar0 · 2026-09-04
• [source] TRACES: new benchmark scores AI agents on unconfirmed scientific discoveries — dair_ai · 2026-09-04
• Apodex launches TRACES, a benchmark grading AI's reasoning trajectories on unsolved science problems — rohanpaul_ai · 2026-09-04
• TRACES agent benchmark grades live execution loops, not answers — SucceededMind · 2026-09-05

1 near-duplicate retellings: omarsar0