Ex-Meta researcher: internal code valsets without deliberate decontamination aren't valid tests
alexpolozov · x · 2026-11-15
alexpolozov argues that treating an internal codebase as "held out" without deliberate multi-tier decontamination makes reported perplexity an invalid generalization test. He notes the dilemma: detailing your decontamination algorithm to enable verifiable public comparison would leak your alpha — so you can't win publicly either way.
More from Models
- Insider rumor: GPT-6.2 delay unexplained as OpenAI's internal-best vs public model gap widens — ChrisGPT · 2026-11-15
- Third-party tests: Thinking Machines' Inkling 2 Small outperforms full Inkling 1 on real workloads — dbreunig · 2026-11-15
- Claim: Gemini 4 Trained Entirely on Google's Own TPUs, No Nvidia Involved — gaganghotra_ · 2026-11-15
- EmbeddingGemma 2 topped HuggingFace trending for a week, a spot usually held by LLMs — gabriberton · 2026-11-15
- Jason Wei: Authoring a TerminalBench problem may soon outrank a Nature paper — _jasonwei · 2026-11-15
- Grok's Imagine Tab Isn't Dead - It's Hidden for New and Inactive Users — nima_owji · 2026-11-15