Dev builds a Forth interpreter into Nemotron 30B weights; LoRA lifts score from 79.1% to 92.1%

Mr_Finious · reddit · 2026-11-07

Inspired by Qwenputer, a developer embedded an untrained Forth interpreter — a 27-layer network whose weights are computed by a Python script — alongside NVIDIA Nemotron 3.5 Lightning (30B total, 3B active) inside the same model. The model writes Forth blocks as plain text; the interpreter executes each word one step per forward pass, with no tool calls and no generation interruption.

Key details:

RPN suits left-to-right generation: words end at spaces so they execute immediately, operands hit the stack before operators; the stack lives in Mamba-2 state, one 20-digit row per entry.
Exactness: no value exceeds bf16's exact-integer limit of 256, every Mamba decay is exactly 1, and 982 reserved tokens carry internal steps; all 765,734 machine tokens matched a Python reference.
The model was harder than the machine: stock Lightning distrusted correct machine output and resorted to hand long division; LoRA fine-tuning raised a 600-problem held-out score from 79.1% to 92.1%.

Results roughly tie stock Lightning on Qwenputer/DeepMind arithmetic sets and lose to Python on long integers (20-digit cap). Explicitly a research project — each machine step costs a full forward pass. The MIT-licensed repo includes the kernel builder with parity tests, a vLLM plugin for the hybrid Mamba/MoE model, LoRA recipes and data, NVFP4/GGUF/MLX builds, and notes on what didn't work, including RL.

Original post →

Dev builds a Forth interpreter into Nemotron 30B weights; LoRA lifts score from 79.1% to 92.1%

More from coding & agent