Circuit view of deep learning: ML is program search with an epiplexity prior
willcb · x · 2026-10-16
will cb argues modern deep learning is less mysterious than it looks:
- Well-regularized ML is essentially well-behaved program search over data and objectives with an epiplexity prior — "we aesthetically wanted it to be more magical than it is."
- He finds the superposition framing unconvincing; his mental model comes from boolean circuits, where grokking is essentially recursive generalization. Backdoors are plausible but don't emerge without adversarial data.
- Bigger models are more sample-efficient because richer existing subcircuits compose into emergent ones, which can then be distilled back into small models via logit-level training.
- Circuits are expressible in bits: fp4→fp32 barely hurts expressivity, mostly affecting optimization dynamics; RL finds high-order bits first.
Related event: Researcher Argues Circuit View Dissolves Deep Learning Mysteries(4 posts)→
More from Research
- Ex-OpenAI researcher claims most neurons are legible now; critics call it hype — anshulkundaje · 2026-10-16
- Physicist pushes back on ML community's acceptance that models may be beyond understanding — tszzl · 2026-10-16
- Survey: AI agent evaluation is getting realistic but still fails to measure safety and reliability — burkov · 2026-10-16
- COVA-SFT dataset trains models to solve text problems with visual chain-of-thought — yoavartzi · 2026-10-16
- USTC paper uses speculative sampling to cut latency in interactive video generation — ustc · 2026-10-16
- Tsinghua's CoTranslator distills reverse CoT to boost 4B model math scores — Tsinghua · 2026-10-16