A quarter of Gemini 4's ML R&D benchmark tasks were unpassable, audit finds
davidmanheim · x · 2026-10-19
David Manheim highlights Simon Mendelsohn's LessWrong analysis of Google DeepMind's Gemini 4 Argon safety report: the report rules out its "AI automating AI research" thresholds (the ones most relevant to RSI) using a benchmark whose own audit admits about a quarter of scored tasks were unpassable — buried on page 67.
Key context:
- Sergey Brin's roundtable comments suggest Gemini is "starting to get onto that RSI treadmill," with researchers and the model working hand in hand
- Some evaluations ran on checkpoints just 10 and 6 days before the final model — rushed conditions meant even top engineers couldn't ensure task quality
- Neel Nanda still calls it Google's most thorough safety report, praising the loss-of-control, alignment, and monitorability assessments
- Additional concerns raised: evaluation validity may worsen with model capability (evaluation awareness + buggy benchmarks produce false negatives for dangerous capabilities); the safety architecture relies on increasingly fragile observability; threshold-based safety cases systematically miss risks below catastrophic thresholds from combinations of partial capabilities
Bottom line: the report demonstrates mitigations work in tested scenarios, but an adequately conservative upper bound on residual risk remains far less established.
More from AGI Musings
- Viral theory claims RSI-capable models can't be open-sourced, DeepSeek dates are frozen snapshots — teortaxesTex · 2026-10-19
- Victor Taelin speculates on AGI-era stroke first aid: recovery falls from 72% to 40% within 6 hours — louisvarge · 2026-10-19
- ICLR reviewer slams AI safety papers as CV padding; critic says Goodharting wrecked the field — JacquesThibs · 2026-10-19
- Anthropic reveals Claude now drives 26% of its own AI R&D — emmanuelvivier · 2026-10-19
- Stanford Opens Free CS336 Course: 18 Lectures on Building LLMs from Scratch — mdancho84 · 2026-10-19
- Designer Pitches Herself Publicly With 'Design Is Dead' Film Amid AI Anxiety — NirantK · 2026-10-19