Perplexity's pitfalls: compute dependence, benchmaxxing, and Anthropic's METR redactions
eliebakouch · x · 2026-11-23
eliebakouch calls OpenAI's pretraining transparency blog extremely important for understanding progress toward RSI, adding two points: (1) raw perplexity numbers depend heavily on compute—efficiency gains are a better metric—and can be benchmaxxed via data mixture; algorithmic progress should be separated from data, which matters less since labs' pretraining data is similar and unlikely to be gamed on held-out sets. (2) The timing is notable: Anthropic redacted all pretraining metrics from the METR evals report.
More from AGI Musings
- Kevin Roose interviewed 150 people for The AGI Chronicles, digging inside AI labs — every · 2026-11-23
- Should the most safety-minded lab stop working first? An AI doom strategy debate — akbirthko · 2026-11-23
- AI Writing Wins Contests: Bad Models, Bad Writers, or Bad Readers? — TuhinChakr · 2026-11-23
- The New Yorker: A.I. Is Solving Decades-Old Math Problems, With Terence Tao as Guide — EricTopol · 2026-11-23
- Perelman solving Poincaré is the math equivalent of Lee Sedol taking a game off AlphaGo — gordic_aleksa · 2026-11-23
- Mathematician proposes making local models great at Lean theorem proving — StefanoGogioso · 2026-11-23