Hashed n-gram value embeddings push GPT to 3.2115 loss, beating 30M track record
RobertTLange · x · 2026-10-27
RobertTLange merged a PR into qlabs-eng/noise setting a new Track 3 (30M) record: a 12-layer × 1024-dim GPT (613.9M params, 402.7M in lookup tables) using hashed n-gram value embeddings, trained by Cut-lite with input-key probes, reaching 3.21149 loss on seed 0 — 0.198 below the previous record of 3.4095.
- Training took 10,604.9s (2h57m) including a 30s end merge, on a single 8xH100 node
- Reproduces with the repo's default command, no extra flags
- For scale: backprop on the fixed 8×512 model hits 3.419 with Adam and 3.340 with ANVIL II (same data, one epoch)
More from Research
- World models matter even without realtime: closed loops and persistent memory — cocktailpeanut · 2026-10-27
- EMNLP paper probes why some LLM biases survive rewording while others don't — radamihalcea · 2026-10-27
- MS-Human-700: open-source human model with 700 muscle actuators lands in MuJoCo — yisongyue · 2026-10-27
- Datology webinar: Curation Studio matched top data baselines with under 20% compute — josh_wills · 2026-10-27
- KANLab: an interactive browser playground for Kolmogorov-Arnold Networks — Lukas Niehaus · 2026-10-27
- CMU's LLM2Cert synthesizes stability certificates verified via SMT and Lean — CarnegieMellonU · 2026-10-27