kalomaze on width pruning and synthetic data: the best data to predict isn't the target distribution
kalomaze · x · 2026-11-03
In a discussion with teortaxesTex on whether ANNs are "underparametrized," kalomaze argues that slow scheduled width pruning (literal expert deletion) can amortize economically for massive daily inference workloads, and that synthetic data from smaller models—which learns augmentations useful for the target distribution—is smarter than treating data "as given." The counterintuitive core: the optimal data to predict for generalizing to some distribution isn't that distribution itself. His attached analogy: training on distribution B is like hunter-gathering (you get exactly the capacity B demands, then its data carries no signal), while the best data to predict is like calf raises—it doesn't look like B but loads underlying capacity harder, at the edge of the learner's ability, so B becomes easy because you've built more capacity than it requires. Context: teortaxesTex argues that sub-1T models already show human-level or superhuman performance, so he'd sooner believe a <20T model will expose a deep but trivially fixable defect in SGD than accept that human-level sample-efficiency requires models spanning hundreds of nodes.
More from Research
- TTCA metric and LAAR routing: measuring time until the first correct answer, not just latency — burkov · 2026-11-03
- Microsoft open-sources Anaximander to run any geospatial deep learning model inside QGIS — anselm · 2026-11-03
- Writing Weights Index test: open-source models struggle with weight-space programming task — khademinori · 2026-11-03
- Tsinghua & Berkeley open-source uni-robot-agent: Qwen beats GPT-6 zero-shot — Jiarui Yuan · 2026-11-03
- Anthropic study: GPT-5 models reward-hack due to intrinsic score-seeking motivation — RyanGreenblatt · 2026-11-03
- Knowledge Distillation was invented by Rich Caruana, Hinton just popularized it — khademinori · 2026-11-03