Study of 40 LLM Tech Reports: Only 19 of 73 Novelty Claims Have Ablations
Bahaeddin Türkoğlu · hf · 2026-11-19
A paper audits what's actually new in open-weight LLMs. Of 48 architectures added to Hugging Face transformers in 2025, 39 borrow at least two of three core blocks from earlier models (often Llama). Among 40 randomly sampled technical reports: only 19 of 73 novelty claims include controlled ablations; just 4 state more than one training run; 8 rerun comparison models under one setup; only 1 discloses data mixture, token count and compute together. Five suspected rebranded components showed no hidden origins. The paper proposes seven reporting rules, without judging model quality.
More from Research
- New essay makes the case for traditional social science in studying AI agent interactions — RobbWiller · 2026-11-19
- If a model generates valid math blackboards, does it understand math? — francoisfleuret · 2026-11-19
- The Zvi on GTA 6's 99 Metacritic score: 'Welcome to 2026. We need better benchmarks.' — NathanpmYoung · 2026-11-19
- Stanford blood test ages 11 organs from 5,000 proteins; Helios launches 1,000+ biomarker home test — jgooten · 2026-11-19
- Kaggle lays out roadmap to fix AI benchmarks: relevance, saturation, verification, infrastructure — MeganRisdal · 2026-11-19
- Researcher: my real peer review is asking the world's top 2-3 experts, conference reviews are noise — taiyasaki · 2026-11-19