Local LLM inference speeds jump ~10x in a week: single consumer GPU now hits 2200 prefill
oran_ge · x · 2026-10-03
Quoting @ivanalogcom's hands-on report: a week ago, dual RTX 5070 Ti cards locally running Qwen Flash managed only 200 prefill/10 decode tokens. Today, with the Strata architecture and a custom PR, a single card reaches 2200/67 — a near-10x speedup on both read and write, a level previously reserved for RTX 5090, DGX Spark, or M3 Ultra rigs, and beyond most model APIs.
- RTX 4090/5090 also see big gains; 5090 with streaming is approaching RTX 6000 speeds (VRAM remains the limit)
- The surge is driving consumer hardware price hikes — even the obscure Intel B70 32G card has sold out
- The gains come from DeepSeek/Qwen Flash's design (splitting intelligence into attention, experts, and lookup) plus Strata's pipelined architecture that stages different capabilities across storage tiers
The author argues this is a huge win for local inference, possibly a converged optimum for current hardware — and a potential death sentence for businesses selling mediocre model compute via API.
More from Infra
- Gridline launches one-API fine-tuning, distillation and batch inference for open models — Scobleizer · 2026-11-19
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19