TWIG: weights as shared discrete decision paths unify pruning, quantization, and adder-based circuits
ITMO · hf · 2026-10-26
TWIG (Tree-structured Weights with Inherited Gates) from LEAD Lab, ITMO University, represents every neural network weight as the end of a path of small discrete decisions: a sign, a shift by a power of two, a zero gate, and an index into a shared table. Weights share upper path segments, so each decision is stored and applied once.
Three consequences: (1) any trained network can be written as a TWIG, and with power-of-two decisions weight layers become circuits of mostly wires and adders, suited to chips with weights fixed at fabrication; (2) pruning, quantization, low-rank factorization, codebooks, and weight sharing all become edits of one object, letting a single optimizer choose across all of them; (3) choosing the decisions is framed as a ground-state problem amenable to mature solvers.
More from Infra
- RTX 4090 workstation vs AI subscriptions: an editor's break-even math — iamthecarguy1234 · 2026-10-26
- Cerebras CS-4 teardown: 2x bandwidth and power, 50% fewer parts vs CS-3 — jwt0625 · 2026-10-26
- theo doubts Windows ARM: porting 20 years of tech debt vs Intel improving — wateriscoding · 2026-10-26
- TokenSpeed's Three-Language Stack: C++ for Correctness, Rust for Agents, Python for Glue — zhyncs42 · 2026-10-26
- Speculative Decoding Explained: a 30% Wrong Guesser Never Changes the Output — anbarth · 2026-10-26
- You Can Train a 1B Model in 3 Weeks: Dev Shares 15.9-Day Training Run Log — arthurcolle · 2026-10-26