Bonsai quant hits 50 tok/s at 128k context on a 24GB card, letting users run two sessions at once
julianharris · x · 2026-09-18
julianharris reports that the Bonsai quant is faster and smaller than his favorite Qwen quant: 50 tok/s at 128k fully loaded on a 4090/24GB (vs 32 before) — fast enough to run TWO sessions at once on one card. He expects roughly 25 tok/s at 256k filled context. Local long-context inference on consumer GPUs has improved significantly.
Related event: Bonsai Quantized Model Hits 50 tok/s with 128k Context on a 24GB GPU(2 posts)→
More from Infra
- Fire at Taiwan Panasonic PCB materials plant may tighten CCL supply further — zephyr_z9 · 2026-11-04
- Same Claude model served by Bedrock one day and Anthropic the next, dev finds — georgos_pappas · 2026-11-04
- Self-improving software is inevitable: closing the agent-observability loop — bibryam · 2026-11-04
- Gigabyte's AI TOP motherboard packs 2TB RAM and 9 GPUs for home AI labs — vincent_koc · 2026-11-04
- Building a 7-GPU 132GB VRAM local inference rig: consolidate on EPYC or split across two systems? — infectiousstupidity · 2026-11-04
- imec neural probe integrates photonics directly into 130nm CMOS backend — jwt0625 · 2026-11-04