A 4M-parameter pixel transformer learns a tiny physics game for about $10
mathemagic1an · x · 2026-07-29
The author describes a toy physics simulation—collisions, wall bounces, and a goal to score in—and trains a small pixel transformer to predict the next frame.
- Model size: about 4M parameters.
- Training compute: roughly $10.
- Despite the tiny budget, the model performs surprisingly well.
- The post points to sampled hallucinations as evidence that the model has learned useful structure rather than trivial pixel copying.
Related event: 4M-Parameter Pixel Transformer Emerges as Physical World Model(10 posts)→
More from Research
- SimpleMark: multi-bit LLM watermark bounds per-token distortion with f-divergence budget — rishabh16_ · 2026-10-16
- Stanford paper: generative representation learning cuts tokenizer training compute by 7-8x — HirokatuKataoka · 2026-10-16
- Cohere Labs to present confidence estimation for reasoning models at EMNLP 2026 — Cohere_Labs · 2026-10-16
- Vals RSI Index extrapolates Google models hitting 130% autonomous AI R&D by Oct 2027 — HaydnBelfield · 2026-10-16
- SpatialClaw Pulled Into Agentic Vision Data; Filtering and Evals Were the Real Lift — CMHungSteven · 2026-10-16
- Visual Lean: drag-and-drop GUI for the Natural Numbers Game runs Lean 4 in WASM — MilesCranmer · 2026-10-16