Hand-crafted 16-neuron net matches trained model (96.0% vs 95.3%), Hilton proposes faithfulness metrics
JacobHHilton · x · 2026-10-18
A community reverse-engineering project drew in Jacob Hilton (ex-OpenAI superalignment). DragggonZ claims to have identified each neuron's role in a 16-neuron network and rebuilt it by hand with Claude's help, no training needed: 96.0% accuracy vs 95.3% for the ARC-trained original, with weight maps relabelled and rescaled to show the same function. Hilton suggests quantitatively measuring mechanistic faithfulness: propagate a probabilistic hidden-state model using each neuron's identified function to produce a mechanistic estimate of the original's accuracy, treating ideal-vs-actual weight discrepancies as noise — and even mechanistically predicting accuracy degradation under added weight noise like N(0, 0.0025²), which typically costs 1-10 accuracy points.
Related event: Researcher Hand-Rebuilds 16-Neuron Network, Beating Original Accuracy(3 posts)→
More from Research
- Ropedia Super Week: world model W0, dexterous bimanual controller DONIC and more — liuziwei7 · 2026-10-18
- 130K-Row RAG Claim Verification Dataset Released on Hugging Face — MaziyarPanahi · 2026-10-18
- Closed AI evals face criticism: opaque, single point of failure, and potential conflicts of interest — evijit · 2026-10-18
- Higher bar for LLM introspection claims: talk at NYU metacognition workshop — tallinzen · 2026-10-18
- Reviewers Complain AI-Generated Papers Waste Disproportionate Review Time — anmarasovic · 2026-10-18
- Label-free optimal transport couplings push CFG flows to 1.91 FID on ImageNet-256 — serrjoa · 2026-10-18