CPU-native LLM pilot passes 4 of 5 gates, but cross-tokenizer distillation still loses
WildPino25 · reddit · 2026-07-21
The author ran a full pilot of a CPU-native LLM recipe on a free Kaggle T4 at 30M parameters, with five pre-registered gates written to git before launch.
- Four gates passed: ternary quantization survived, the pipeline resumed across a dozen session kills, multi-GPU scaling reached 1.80×, and long-context recall stayed stable.
- The favorite idea failed: cross-tokenizer distillation from a teacher with a different vocabulary kept only about 15% of the teacher’s information naively, and a chain-rule fix recovered 83–86% of it at zero extra compute.
- Even with that fix, plain cross-entropy still won at every stage.
- The author notes an important confound: the teacher is a code model scoring children’s stories, so this is an off-domain cross-tokenizer distillation test, not proof that distillation is dead.
- An MoE upcycle appears to cost the distillation arms +0.054 BPB with no recovery, which is now a separate investigation.
- Next up is a first real rung: a 30M total / 11M active model trained on Python, with public pre-registration and strict evaluation rules.
More from Research
- An interactive Zarr explainer shows how AI is changing technical education — MaxLenormand · 2026-07-21
- 3D-Fit finds LLMs can handle multiple molecular constraints, but still lag diffusion models — insilicomedicine · 2026-07-21
- Open-AoE opens 2,000 hours of egocentric manipulation video for robot learning — inclusionAI · 2026-07-21
- Cisco releases Antares small models to localize code vulnerabilities — aminkarbasi · 2026-07-21
- New paper studies how to detect memorization in autoregressive language models — rvp · 2026-07-21
- Apple-π benchmark asks whether video models reason about physical laws or just mimic motion — liuziwei7 · 2026-07-21