Code perplexity evals now measure LLMs predicting their own generated code
DZhang50 · x · 2026-11-14
Responding to khoomeik's thread on whether Ant's cross-entropy gains come from data or algorithms, DZhang50 flags a measurement contamination problem: many teams now measure code perplexity on "held-out" internal codebases that are themselves LLM-generated, so they're effectively measuring an LLM's ability to predict its own code.
khoomeik argued a superexponential CE decline is a much weaker update for algorithmic progress unless data effects are ablated; synthetic-data-driven intelligence explosion (cf. RL) is possible but usually not what "algorithmic progress" means.
More from AGI Musings
- Open weights are yours once downloaded, argues X post: no one can retake or censor them — alex_verem · 2026-11-14
- Azeem Azhar sketches the cost of 'manufacturing' a worker: AI vs human economics — Exponential View (Azeem Azhar) · 2026-11-14
- Will anyone actually learn from the AI air traffic control outage panic? — sebkrier · 2026-11-14
- Reddit devs debate AI-driven CS job crash: grad jobless 6 months, 'only the exceptional get hired' — ghumare64 · 2026-11-14
- Scott Aaronson: grad students scared after OpenAI's release of 370 math proofs — rohanpaul_ai · 2026-11-14
- Reading the Jensen Huang firestorm through René Girard's scapegoat theory — itsOmSarraf_ · 2026-11-14