Local LLM inference speeds jump ~10x in a week: single consumer GPU now hits 2200 prefill

oran_ge · x · 2026-10-03

Quoting @ivanalogcom's hands-on report: a week ago, dual RTX 5070 Ti cards locally running Qwen Flash managed only 200 prefill/10 decode tokens. Today, with the Strata architecture and a custom PR, a single card reaches 2200/67 — a near-10x speedup on both read and write, a level previously reserved for RTX 5090, DGX Spark, or M3 Ultra rigs, and beyond most model APIs.

RTX 4090/5090 also see big gains; 5090 with streaming is approaching RTX 6000 speeds (VRAM remains the limit)
The surge is driving consumer hardware price hikes — even the obscure Intel B70 32G card has sold out
The gains come from DeepSeek/Qwen Flash's design (splitting intelligence into attention, experts, and lookup) plus Strata's pipelined architecture that stages different capabilities across storage tiers

The author argues this is a huge win for local inference, possibly a converged optimum for current hardware — and a potential death sentence for businesses selling mediocre model compute via API.

Original post →

Local LLM inference speeds jump ~10x in a week: single consumer GPU now hits 2200 prefill

More from Infra