Textbook author: 99.9% accuracy can mean zero scientific discoveries
bravo_abad · x · 2026-09-20
A textbook author for science students argues that a good architecture cannot rescue a leaky split, a biased label, or a metric that doesn't match the question. Ideas usually filed under "technical details" are definitely not details:
- Your test split defines your scientific claim
- "Ground truth" is rarely ground truth
- 99.9% accuracy can mean zero discoveries
- Your loss function is a scientific assumption
- A prediction without uncertainty is often only half a prediction
The author collected 11 such pitfalls, with figures, in a new post on Discovery at Scale.
Related event: ML Textbook Author: 99.9% Accuracy May Mean Zero Discoveries(2 posts)→
More from Research
- Microsoft researcher points out BERT was already called an LLM back in 2019 — JFPuget · 2026-09-20
- Have we seen an acceleration in discoveries? Cyber spikes, math rises, algorithms flat — soumitrashukla9 · 2026-09-20
- TovanaEngine: a local world model trained on 50K+ real SWE-bench coding-agent runs — Decent-Ad9950 · 2026-09-20
- After Chess and Go: Can Any AI Engine Actually Beat Humans at Scrabble? — zuilserip · 2026-09-20
- rasbt: Jev's Secret Sauce Is Data, Not the Algorithm — Laya Rival Falls Far Short — RichmanRonald · 2026-09-20
- Sebastian Raschka open-sources an end-to-end 'AI text detector from scratch' project — rasbt · 2026-09-20