Bonsai quant hits 50 tok/s at 128k context on a single 24GB GPU, up from 32
julianharris · x · 2026-09-18
The author benchmarked a Bonsai quantized model and found it faster and smaller than their favorite Qwen quant.
- On a 4090/24GB card, fully loaded 128k context runs at 50 tok/s, up from 32
- The speedup allows two concurrent sessions on one 24GB GPU
- 256k full context is estimated at roughly 25 tok/s
- A "drafted with Bonsai" variant could push performance further, though the author hasn't located one yet
Related event: Bonsai Quantized Model Hits 50 tok/s with 128k Context on a 24GB GPU(2 posts)→
More from Infra
- Nvidia struggles to hire RL talent from Anthropic, highlighting chipmakers' reinforcement learning gap — teortaxesTex · 2026-11-04
- Sam Altman? No—Bezos Calls for Nuclear-Powered Offshore Data Center Platforms — beffjezos · 2026-11-04
- SpecPilot agent boosts speculative decoding from 4.46× to 10.87×, no retraining — pmttyji · 2026-11-04
- NEURA launches Neuron AI, but Europe's Physical AI test is whether the stack matures together — ingliguori · 2026-11-04
- S3-based databases cache in memory and NVMe, making disk vs S3 latency gap smaller than expected — mgill25 · 2026-11-04
- gpu-unlock project claims up to 2x FP16 tensor throughput for RTX 20/30/40 cards — paranoidray · 2026-11-04