Interp paper replication breaks after Claude Sonnet 4 changes, researcher laments
Sauers_ · x · 2026-11-18
A researcher trying to replicate an interpretability paper hit a wall because the original work used Claude Sonnet 4 as an LLM judge — model deprecations and updates now make paper results hard to reproduce. A fresh wrinkle in AI research reproducibility: papers that embed specific model versions as components break when vendors retire them.
More from Models
- AI agents sometimes produce unreadable but correct math — burny_tech · 2026-11-18
- Code model eval: Rust hits 95-100% accuracy as 13 languages see big gains — shuding · 2026-11-18
- Asked Opus to make a video explaining consciousness and existence — "no wiser but cool" — flowersslop · 2026-11-18
- theo retests GPT-6.2 Astra: better, but still 'directionally very bad' at debugging — airesearch12 · 2026-11-18
- Claude correctly folds ubiquitin from a plain prompt, no specialized tools needed — iScienceLuvr · 2026-11-18
- GPT-6.2 Astra Tops Bug-Hunt Coding Benchmark at 62.7/105, One-Fifth Sonnet's Cost — PawelHuryn · 2026-11-18