Should labs run adversarial evals to detect models deceiving them about AI R&D?
dfrsrchtwts · x · 2026-11-08
dfrsrchtwts proposes an AI governance idea in a discussion with Charlie Bullock and Neil Chilson: labs should run adversarial evaluations testing whether their models could fool them about AI R&D — i.e., whether automated AI R&D could deliberately produce models with certain propensities or capability profiles undetected.
He admits such evaluations aren't yet well-understood. He also weighs two regulatory options: either allow humans to just press 'y' as AIs conduct all R&D, or mandate that some share of inference compute serve real customers — a rule he suspects market-oriented voices would dislike, and which may be hard to tailor well.
More from AGI Musings
- AI capability growth as a tug of war: recursive self-improvement vs diminishing returns — CFGeek · 2026-11-08
- Stanford prof: AI-accelerated diagnosis is low-hanging fruit, not the real bottleneck — anshulkundaje · 2026-11-08
- Chollet: Scaling Is Only a Multiplier, Smarter Methods Scale Better — fchollet · 2026-11-08
- Kundaje: AI-Accelerated Diagnosis Is Low-Hanging Fruit, Not the Bottleneck — anshulkundaje · 2026-11-08
- One Word in the Bitter Lesson Misled Silicon Valley on LLMs — sebkrier · 2026-11-08
- AI Alignment Discourse Through the Years: From Doom to a Suboptimal Shade of Blossom — voooooogel · 2026-11-08