antirez demos his OSS inference project: README too small to hit fastest spot of curve
antirez · x · 2026-11-08
Replying to steeve's question about scaling with many parallel sessions, antirez notes there's no immediate way to test that, but shows a "quite compelling" demo video of his open-source inference project. He points out the README file isn't big enough to trigger the fastest spot of the performance curve, hinting more headroom on larger inputs. In the same thread he explains that large batches switch to layer-major inference.
Related event: antirez Explains Local Inference Scaling Under Larger Batches(4 posts)→
More from Infra
- Chamath: Compute Spot Prices Doubling Would Force SpaceX to Reprice Anthropic Deal via 90-Day Exit — rohanpaul_ai · 2026-11-08
- Users are buying RTX 5060 Ti 16GB cards en masse for local AI — ab2377 · 2026-11-08
- Study: data centers' impact on residential electricity prices is only marginal — AccBalanced · 2026-11-08
- RustingStrata hits 160 tok/s on RTX 3060 running 35B MoE, ~2x faster than llama.cpp — Serious_Scarcity7818 · 2026-11-08
- Geek's Throne wishlist gets a $5M exabox (256x MI300X) fully funded — gnukeith · 2026-11-08
- Dynatrace Closes $915M Arize Deal as 1,100+ Customers Run Its Agents — shashib · 2026-11-08