Chris Potts critiques auto-interp evaluation: precision, recall, and the case for ex ante testing
ChrisGPotts · x · 2026-10-19
Chris Potts argues the auto-interp field has persistently ignored precision and recall, over-relying on heuristic spot-checks that invite just-so stories. His extended evaluation of Bills et al. (2023) advocates an intervention-based, ex ante evaluation approach; collaborator Leo's new LessWrong post suggests sympathy with that assessment.
More from Models
- Bittensor subnet Ingot beats Qwen's own FP8 quantization within 4 days — const_reborn · 2026-10-19
- Cohere merges llama.cpp support, all its latest model architectures now compatible — nick_frosst · 2026-10-19
- Liquid AI explains designing LFMs for phones: less compute, better architecture and data — JosephJacks_ · 2026-10-19
- Report: Fable 5.5, Gemini 4 Carbon and GPT-6.1 Terra all due this month — Angaisb_ · 2026-10-19
- Model saturation data is biased: vendor harnesses muddy capability conclusions — matt_slotnick · 2026-10-19
- Claude recovered a man's hard drive, then refused to find his own $8,000 wallet seed phrase — MoonL88537 · 2026-10-19