antirez: large batches switch to layer-major inference, nearly as fast as full-resident
antirez · x · 2026-11-08
Answering steeve's question about batch-size>1 scaling, antirez (Salvatore Sanfilippo) explains that the low prefill in his demo video was due to a short prompt on a single-token path — the opposite of the concern raised: large prefills or large batches switch to layer-major inference and run almost as fast as full-resident weights. The project is OSS, so anyone can try it.
He adds that many local-inference use cases, especially non-resident ones, are effectively single-user scenarios with tools like pi or opencode attached to a local server, and notes the README file isn't big enough to hit the fastest spot of the performance curve.
Related event: antirez Explains Local Inference Scaling Under Larger Batches(4 posts)→
More from Infra
- Musk endorses orbital AI compute pitch: 100 GW/year could add $1T annual revenue to SpaceX — elonmusk · 2026-11-08
- BF16 GPU TFLOPs comparison table proves its value over time, says HF dev Zach Mueller — TheZachMueller · 2026-11-08
- Starlink adds 1.6M consumer subscribers in a quarter, hits 13.6M worldwide — XFreeze · 2026-11-08
- Leeds AI datacentre row: water use equals just 5 car washes, 1,000+ jobs at stake — danbri · 2026-11-08
- Solar compiler compiles in 4.8s vs solc's 2h, cuts fuzzing RAM 4.7x in Echidna test — banteg · 2026-11-08
- Micron's buyback now retires 52% more shares than a month ago as memory prices dip — AntDX316 · 2026-11-08