Indie Dev Builds Training Pipeline Aiming at a 2B Model That Does Real Work
cephaloform · x · 2026-11-12
A developer is incrementally building a training pipeline aiming to output a 2B model that does actual work, unconcerned about large batch sizes.
Mid-run he found issues with the RNN training environment and split the pipeline for safety. Work streams include optimizer engineering, curriculum ordering, auxiliary loss design, plus funsearch-style tasks (bin packing, maxcut, sorting networks) — the ones that weren't busted worked reasonably well. A hands-on small-scale pretraining log.
More from Research
- 200-seat UC Berkeley course on full-stack post-training, RL and applied RSI announced — karinanguyen · 2026-11-12
- NVIDIA's Déjà View: 117M looped transformer matches models 10x larger at 3D reconstruction — NVIDIAAI · 2026-11-12
- OpenAI finds models are getting harder to monitor through their chain-of-thought — xiamengzhou · 2026-11-12
- Show Reddit: ge, a Rust GGUF engine, benchmarks against llama.cpp on a 4GB RTX 3050 — Raptor2496 · 2026-11-12
- 1,500 AI-for-Science Papers in a Week Share One Question: Compared With What? — bravo_abad · 2026-11-12
- AI system proves longstanding quantum computing conjecture in three days, Lean-verified — _arohan_ · 2026-11-12