Bixbench3: Frontier Agents Score Below 50% in Reproducing Paper Analysis
xeophon · x · 2026-08-27
Bixbench3 evaluates whether models can reproduce entire analyses from papers. Agents were tasked with end-to-end reproduction, with some runs spending over 1B tokens. Results show frontier agents still score below 50%, failing due to real-world issues like environment misconfigurations, quitting early, or fabricating data.
Related event: BixBench3: Frontier AI Agents Reproduce 48% of Bio Workflows(4 posts)→
More from coding & agent
- Proof launches x401, first protocol letting AI agents prove who they represent — csuwildcat · 2026-10-15
- A versioned bootstrap loader pattern for keeping AI-generated code always shippable — NickPassig · 2026-10-15
- Dev vibe-codes open-source newsletter platform, 83% cheaper than Beehiiv from $19/mo — marclou · 2026-10-15
- ERP AI launches Proto: 'headless SaaS' business software built for AI agents — Aiden_Tech_Ai · 2026-10-15
- Engineering managers need deep reading, not just agent summaries: 5 recommended resources — rseroter · 2026-10-15
- Adding motion graphics with just a prompt: the Claude Motion x ElevenLabs MCP workflow — CurieuxExplorer · 2026-10-15