A CPU Inside an LLM: Zero-Gradient Compilation of Exact Circuits into Pretrained Weights
Tanmay Sah
cs.AI
2026-11-20
Zero-gradient closed-form primitives write exact 6-digit arithmetic into pretrained LLM weights: 100% on Qwen 3.5, no language regression, vs 61.7% for LoRA SFT.
Exact multi-digit arithmetic is a persistent failure mode for autoregressive LMs. The deployed fix is tool routing: pause decoding, hand the expression to a Python interpreter, paste the result back. That costs latency and attack surface, and in air-gapped or pure-tensor inference environments there is no host process to call. The academic alternative, RASP and Tracr, compiles programs into scratch-built toy Transformers that assume biases, absolute positions, and unnormalized wires, assumptions that modern open weights (RMSNorm, bias-free SwiGLU, RoPE, gated linear attention) violate across the board.
This paper performs weight surgery on stock models: write exact 4-8 digit signed add/subtract/multiply circuits directly into the weights of a pretrained 3B-4B LM, with zero gradient steps, zero training examples, zero runtime hooks, and no loss of general language behavior.
Three obstacles, three closed-form primitives.
Unknown scaling. RMSNorm multiplies each token's hidden vector by a coefficient known only at runtime, swinging more than 250×. The paper first proves a single-layer impossibility: a bias-free GLU layer's output collapses as the scale shrinks, so no threshold test is realizable. The two-layer fix rests on an integral identity. Evaluating SwiGLU over an exponentially spaced grid of gate scales turns the unknown multiplier into a horizontal shift along a flat quadrature plateau, and a ratio readout v/r cancels it. With grid step 0.55 the discretization error stays below 6.3×10⁻⁶, and IEEE 754 rounding bounds are carried all the way through.
Cross-token routing. For Granite's RoPE attention, standard positive frequency weights leave positive sidelobes (S(δ) ≥ +0.61) that leak attention onto off-target tokens. Signed Chebyshev weights build a kernel with S(0)=+1 and S(δ) ≤ −0.355 everywhere else, so inactive queries park on their own zero-payload self-sink. Keys and values carry [λ, λ·x] pairs and the readout divides the scale away. Qwen's GatedDeltaNet layers get a different treatment: clamp the forget gate to exactly zero (the layer becomes memoryless) and exploit SiLU(u) − SiLU(−u) ≡ u for exact token shifts.
Placement. The obvious trick, SVD-projecting K wires free inside the original width, provably damages language (a Ky Fan eigenvalue lower bound; measured next-token KL is 43× to 3,800× worse). Instead the construction appends K new wires axis-aligned with zero cross-coupling, rescales original channels by √(d/(d+K)) to absorb the wider RMSNorm denominator, and uses (+254, −256) differential tied-unembedding pairs that cancel bit-exactly to +0.0 on every non-trigger step, leaving base logits untouched.
On top sits a 5-stage declarative compiler that introspects the model config and tokenizer, then synthesizes and schedules the circuit with no manual wire or layer IDs. The circuit fires only on well-formed arithmetic and hands next-token control back to the base LM immediately after the answer.
6-digit signed {+, −, ×}, N=1,200, BF16:
| Configuration | Arithmetic | Language retention |
| Granite 4.2 3B base | 15.67% | PPL 14.30 |
| Granite + this paper | 99.17% (100% with FP32 sidecar) | KL 0.0014; full GSM8K 73.16→74.15% |
| Granite + LoRA (12.8K samples) | 42.2% | 0% inline continuation |
| Qwen 3.5 4B base | 47.33% | PPL 9.47 |
| Qwen + this paper | 100% | KL 0.0007; 14/14 noninferiority splits pass |
| Qwen + LoRA | 61.7% | 0% inline continuation |
The graft touches 1.33M (Granite) and 2.39M (Qwen) non-zero weights, under 0.06% of parameters. Across 11,134 out-of-domain prompts the circuit never fires and the max logit perturbation is exactly 0. Ablations are clean: replacing the Mellin grid with a calibrated single-scale SwiGLU drops arithmetic to 8.3%/20.0%; dropping the √(d/(d+K)) compensation costs 19 to 54 points of ARC-C. A factorized block-sparse sidecar holds A100 memory at 7.89/10.65 GiB versus 13.55/19.09 GiB dense, speeds 2K-token prefill by up to 2.67×, and composes with INT8/INT4 quantization. A 42-layer Gemma 4 E4B audit isolates failure modes: grafts on KV-shared layers fail (47.9%), calibrated grafts on unshared layers reach 100% (64/64).
For air-gapped, regulated, or pure-tensor inference, this is a working path to deterministic computation inside a single checkpoint with no host code. Mechanistically it is a complete recipe for exact digital logic in normalized, real-world Transformers, which weight-space programming and interpretability work can build on.
The security reading matters more than the arithmetic. Zero out-of-domain triggers and KL shifts under 0.002 nats mean a dormant, input-triggered payload grafted into open weights is invisible to behavioral benchmarks and perplexity checks. The author explicitly calls for white-box structural audits of open-weight checkpoints: dimension checks, scans for axis-aligned block decoupling, and SHA-256 provenance.
Honest framing: for most applications tool calls remain the simpler option. This is a proof of concept with methodological value, not a drop-in production technique.
Author-acknowledged. BPE manifold mismatch: a 1-digit emitter dropped Granite GSM8K[:60] from 68.33% to 36.67%, and 18 of 19 regressions had mathematically exact answers; the base model mis-copies token sequences outside its pretraining distribution (64 read back as 6.4), fixed only by a Base-1000 canonical emitter that restores 70%. Scope: the main layout is fixed 6-digit with no division (division needs 40 layers, or 8 layers at 100% inside an FP32 sidecar); unbounded inputs and halting loops exceed constant-depth TC0. Precision: BF16 worst-case carry bounds formally hold only for smaller operands; 1-ULP stability on the upper range is an empirical observation with a 1% flip rate, and 100% relies on an FP32 sidecar.
Reader-side concerns. Single independent author, no peer review, all numbers self-reported; the companion scripts verify constants and floating-point instances, not the theorems. Each model family needed its own hand-derived primitives, so the compiler automates scheduling rather than architectural generalization. Full GSM8K moved +0.99 points, within the noninferiority margin: the circuit fixes arithmetic without demonstrating a reasoning gain. The LoRA baseline trained for 400 steps on 12.8K examples, and whether it was tuned to parity is not established.