A CPU Inside an LLM: zero-gradient compilation of exact arithmetic circuits into pretrained weights
Tanmay Sah · hf · 2026-11-20
This paper introduces closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights — zero training examples, zero runtime hooks — while bounding per-layer normalization drift and preserving natural-language competence.
Three analytical mechanisms:
- Two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown RMSNorm scaling into translation along a log-spaced gate grid (>250× dynamic range) with IEEE 754 error bounds
- Cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts
- Axis-aligned width extension (d→d+K), Base-1000 canonical BPE emission, differential tied unembeddings
Results: an automated 5-stage compiler synthesizes 4–8 digit signed {+,-,×} circuits in constant TC^0 depth. On 6-digit arithmetic, IBM Granite 4.2 3B reaches 99.17% and Qwen 3.5 4B 100.00% ALU-prefix accuracy (100% with an FP32 sidecar), with 98–100% mid-sentence resumption — far outperforming 12.8K-sample LoRA (42.2%/61.7%). A block-sparse sidecar keeps A100 memory at 7.89/10.65 GiB. Boolean and 5-bit Majority/Parity circuits pass at 100%.
More from Models
- GPTZero scans millions of citations: hallucination rates flat or rising at CVPR, ICLR, EMNLP — HanchungLee · 2026-11-21
- Investor slams per-token pricing: verification burns millions of tokens, calls for pay-per-outcome — jasondeanlee · 2026-11-21
- 4,700-vote marketing eval: GPT-6.2 Sol beats Opus 5.5 at 58% win rate and a quarter of the cost — boringmarketer · 2026-11-21
- Leaked model codename lineage: Neon, Carbon, Argon, Sodium traced to Hydrogen and Helium — dejanseo · 2026-11-21
- Blogger: OpenAI model's Erdős unit distance result was 'priced in' long ago — JacquesThibs · 2026-11-21
- OpenAI expands Daybreak security model access for defenders and opens GPT-6 Astra Blue alpha — OpenAIDevs · 2026-11-21