steeve asks antirez how his inference project scales with batch-size>1
steeve · x · 2026-11-08
steeve, "hyped up" by antirez's demo, asks how the open-source inference project scales with batch-size>1. antirez's answer — low prefill was due to a short single-token-path prompt; large batches switch to layer-major inference at near full-resident speed — is covered in the same thread. This post is the quoted repost of that exchange.
Related event: antirez Explains Local Inference Scaling Under Larger Batches(4 posts)→
More from Infra
- Chamath: Compute Spot Prices Doubling Would Force SpaceX to Reprice Anthropic Deal via 90-Day Exit — rohanpaul_ai · 2026-11-08
- Study: data centers' impact on residential electricity prices is only marginal — AccBalanced · 2026-11-08
- RustingStrata hits 160 tok/s on RTX 3060 running 35B MoE, ~2x faster than llama.cpp — Serious_Scarcity7818 · 2026-11-08
- Geek's Throne wishlist gets a $5M exabox (256x MI300X) fully funded — gnukeith · 2026-11-08
- Dynatrace Closes $915M Arize Deal as 1,100+ Customers Run Its Agents — shashib · 2026-11-08
- Running Multiple LLM Slots on One 7900 XTX Boosts Total Tokens/Sec — c2btw · 2026-11-08