Inside a 26B Looped Transformer: One Vector Swap Lifts 4th-Pass Accuracy from 34% to 74%

pandeyparul · x · 2026-11-22

DrHB published a deep dive into Thinking Machines' two 26.4B open-weight looped MoE transformers — Inkling 2 Mini (fixed at three passes) and varloop (one to five passes). Five findings: Mini's perplexity oscillates with period three beyond three passes while varloop stays flat up to nine; Mini encodes pass count across hundreds of state coordinates; a single subtractable vector lets Mini stop at two passes (perplexity 18.0→11.1) and adding it enables a fourth pass, boosting four-table lookup accuracy from 34% to 74%; later passes re-read the prompt (hiding lookup tables in passes 2-3 drops accuracy from 98% to 33%, worse than one pass); and confidence-based stopping matches five-pass accuracy in 2.9 passes on puzzles but saves nothing on ordinary text.

Original post →

More from Research

Research channel →