Four frontier models score 97.8–100 on a self-built materials-science research eval
v_aravind · reddit · 2026-11-14
A graphene/2D-materials researcher built a 7-part test mimicking everyday lab work: cleaning tensile data with unit/outlier traps, writing a CIF density function checked on hidden structures, physics estimates, finding 10 planted manuscript errors among 6 decoys, false-premise/fake-paper calibration questions, experiment design, and constrained abstract writing.
Gemini 4 Carbon, Gemini 4 Hydrogen, Claude Opus 5.5 and Claude Fable 5.2 each ran 3 fresh times: means of 100.0/99.7/99.3/98.5, all 12 runs between 97.8 and 100, each taking 7–24 minutes. Every run caught the unit trap, passed hidden CIF tests, nailed subtle physics, found all 10 errors with zero false positives, and refused fabricated premises; lost points were formatting trivia. Three runs found a real unplanted manuscript error.
Author's read: bounded, procedure-known work is saturated, consistent with FrontierScience/METR/PaperBench — the gap is long-horizon open-ended work. Next step is a simulated-lab agent campaign with budgets, instruments, planted surprises and signed action logs over 2–6 hour runs.
More from Research
- Reinventing clinical guidelines in the AI era: expert panels worth watching — zakkohane · 2026-11-14
- Anthropic's TASTE benchmark: best model judges safety proposals at 60%, below human researchers — maksym_andr · 2026-11-14
- Karl Astrom, pioneer of adaptive control and POMDP theory, dies at 92 — gautamcgoel · 2026-11-14
- ELIZA's SPACKS script: the chatbot's earliest poetry-critic role unearthed — begusgasper · 2026-11-14
- Gensyn launches evidence-mining agent contest with $7,250 prize pool — benfielding · 2026-11-14
- Regeneron's baloncibart cuts standing heart rate by 11.4 bpm in Phase 2 POTS trial — ahandvanish · 2026-11-14