Perplexity's pitfalls: compute dependence, benchmaxxing, and Anthropic's METR redactions

eliebakouch · x · 2026-11-23

eliebakouch calls OpenAI's pretraining transparency blog extremely important for understanding progress toward RSI, adding two points: (1) raw perplexity numbers depend heavily on compute—efficiency gains are a better metric—and can be benchmaxxed via data mixture; algorithmic progress should be separated from data, which matters less since labs' pretraining data is similar and unlikely to be gamed on held-out sets. (2) The timing is notable: Anthropic redacted all pretraining metrics from the METR evals report.

Original post →

More from AGI Musings

AGI Musings channel →