Curated Data Beats Compute: 12B Model Matches 30B With a Fifth of the Pretraining FLOPs
bodonoghue85 · x · 2026-10-24
Team Arimorcos reports that data quality is the ultimate compute multiplier: curated data matched their largest baseline with 6x less compute, and a 12B-A1.4B model beat a 30B-A3B model using only a fifth of the pretraining compute. Full walkthrough livestream Tuesday 10am PT with hurrycane, lukemerrick and joshwills.
More from Infra
- Humansand Speeds Up NVFP4 Decode Kernel by Up to 10% While Improving Accuracy — noahdgoodman · 2026-10-24
- Recursive's 40B Flint Hits ~200 tok/s on M3/M5 Ultra via MLX-VLM with Fused Metal Kernels — timshi_ai · 2026-10-24
- US electricity prices up 34% at investor-owned utilities since 2020 vs 19% public — randal_olson · 2026-10-24
- ExecuTorch now builds on Windows Snapdragon X Elite, demoing on-device Llama 3.2 1B — carrycooldude · 2026-10-24
- PyTorch models now run directly on Cortex-M microcontrollers, under 90 KiB RAM — PyTorch · 2026-10-24
- Strata adds Qwen 3.6 35B support, but decode speed loses to llama.cpp on MI50 — JsThiago5 · 2026-10-24