~$11 per billion tokens: decentralized-trained model's inference optimizations revealed
bittingthembits · x · 2026-09-13
jondurbin quotes his team's decentralized training run, priced around $11/b tokens with impressive MFU. But he stresses the real point isn't the training itself — it's the inference optimizations the architecture enables:
- sparse fp4 native compute for routed experts
- fixed KV cache for most attention
- sparse latent for the rest, tiny model weights
Numbers come from an unoptimized vllm branch on their own arch, and he caveats the comparison isn't fully fair. Core claim: limitless tokens on cheap hardware, using a few nodes from liumio.
More from Infra
- Would LITE accepting MRVL's merger offer prove its moat was overstated? — BenBajarin · 2026-10-30
- Claymation body horror on 8GB: three wrong settings, not the GPU, made local renders slow — roro9- · 2026-10-30
- durable-object-migrator: zero-downtime exports for Cloudflare Durable Objects — samgoodwin89 · 2026-10-30
- An unnamed firm already runs a fairly scaled dedicated vehicle for compute lending, says analyst — matt_slotnick · 2026-10-30
- GPU break-even throughput vs open-weight token prices up 25-83% since May — bookwormengr · 2026-10-30
- Sam Altman says the chip supply chain's invisible hand is failing — and DRAM is why — SnoozeDoggyDog · 2026-10-30