MoE hits ~2000 tok/s prefill on 32GB VRAM for bulk text extraction
MkGod · reddit · 2026-09-14
On 2x RTX 5060 Ti (32GB VRAM), the author benchmarked local models for bulk structured extraction: Qwen 27B Q4KM managed 50 t/s, while the MoE Ornith-1.5-35B Q4 reached 2000 t/s prefill and 100-110 t/s generation. For read-heavy, short-output extraction workloads, the prefill jump cuts total time dramatically.
The post also asks whether modern 7B-14B dense models suffice for schema extraction, or whether MoE is the obvious pick, and whether any leaderboard tracks real local prefill/decode speeds.
More from Infra
- Your AI agent would cost 260% more without prompt caching, Arena data shows — arena · 2026-10-30
- Nebius runs auction-driven GPU pricing, lifting rates ~20% amid tight supply to 2028 — BenBajarin · 2026-10-30
- macOS update silently re-adds eMudhra root certs users had removed — steipete · 2026-10-30
- Bittensor's Terza Inference routes one API across miners' models on subnet 103 — const_reborn · 2026-10-30
- SGLang Summit 2026 sells out sponsor booths and Full Passes ahead of Nov 12–13 event — ying11231 · 2026-10-30
- OpenRouter rebuilds provider routing: 46% fewer capacity errors, 16.9% faster TTFT — steipete · 2026-10-30