Inkling 2 Small notes: 594B-param MoE with 17B active, RL ate ~half of pretraining FLOPs
teortaxesTex · x · 2026-11-08
Detailed notes on the Inkling 2 Small release, shared by teortaxesTex as the highest-signal digest.
Architecture
- 48-layer decoder-only transformer, 594B total params, 17B active per token
- Sparse MoE feed-forward backbone: 8 of 384 experts routed per token
- Hybrid local/global attention layers; context up to 1M tokens
- ViT reintroduced, likely for training efficiency
Training budget
- Pretraining: 8.4T + 3.8T = 12.2T tokens; RL: 3T tokens
Joint RL run details
- Each optimizer step: 12 minutes, 2.8B rollout tokens; median staleness 17 updates (3.4h policy lag)
- Mean rollout 170K tokens; p99 rollout 1.5M tokens compacted 5+ times inside a 256K window
- Per-family mean rollout length: coding ≈290K, web/app ≈260K, office ≈250K, STEM ≈185K, search ≈120K, vision ≈37K, tool-use ≈23K, chat/safety ≈4K; coding episodes routinely exceed the window, so compaction was the default
- RL used almost half the FLOPs of the 8.4T pretraining phase at 3× wall-clock
- Effort levels randomly sampled during RL and conditioned on
More from Models
- Open coding models turned telegraphic: filler words down 25–50% across 12 models — thesephist · 2026-11-08
- Claude Opus 5.5 casually suggests deleting your entire ~/.codex/sessions directory — keunwoochoi · 2026-11-08
- Polymarket odds: only 39% chance Claude 6 ships by March 2027 — Polymarket · 2026-11-08
- Law professor asks Claude to update page numbers, returns to find a year of files deleted — Polymarket · 2026-11-08
- Security researcher: GPT-6 cyber devises exploit techniques never seen publicly before — steipete · 2026-11-08
- METR finds Anthropic model progress accelerating 1.9x since Oct 2025; Hanson calls it one-time speedup — sebkrier · 2026-11-08