Bixbench3: Frontier Agents Score Below 50% in Reproducing Paper Analysis
xeophon · x · 2026-08-27
Bixbench3 evaluates whether models can reproduce entire analyses from papers. Agents were tasked with end-to-end reproduction, with some runs spending over 1B tokens. Results show frontier agents still score below 50%, failing due to real-world issues like environment misconfigurations, quitting early, or fabricating data.
More from coding & agent
- Offline Wikipedia RAG via 100-line Bash script works on local LLMs — mantisalt · 2026-08-27
- Hermes LCM: Compresses 173k Tokens to 2.8k with Full Retrievability — iamrobotbear · 2026-08-27
- Google releases Gemini 3.5 Transcribe: filters filler words and understands codebase context — AI_Andrew · 2026-08-27
- Why AI agents need stablecoins: traditional settlement takes T+2 to T+60 days — kleffew94 · 2026-08-27
- Migrated docs to Confluence because Notion Cloudflare blocks AI agents — corbtt · 2026-08-27
- Benchmark: OpenCode degrades open model performance, Pi framework leads — PMinervini · 2026-08-27