Is PPL on internal code still a good pretraining eval when code is AI-generated?
josh_wills · x · 2026-10-22
A thought-provoking eval question: the gold standard of pretraining evals has been perplexity on internal codebases and docs, but in a world where everyone's code is heavily AI-produced, is it still a good held-out metric? The author suggests pre-AI code may find its greatest value as evaluation data.
More from Models
- Daniel Rock: New frontier models react far more aggressively to context instructions — danielrock · 2026-10-22
- Researchers credit OpenAI's Daybreak program with GPT-5.6 access for Chrome, GitLab exploits — Eric_Wallace_ · 2026-10-22
- Zeroing 7 neurons cuts Qwen3-8B sycophancy from 76% to 0% in Opus-assisted experiment — Sauers_ · 2026-10-22
- Zeroing 6 neurons cuts Qwen3-8B sycophancy from 44% to 0%, interpretability charts show — Sauers_ · 2026-10-22
- User confused as Claude Code fast mode burns API credits instead of 20x subscription — RileyRalmuto · 2026-10-22
- Abliterated MiniCPM5-2B-SFT released on Hugging Face, refuses only ~19/20 of the time — XtrComSu · 2026-10-22