Forked Strata hits 7,357 tok/s prefill on IBM AC922 running an 8B Qwen model
okoyl3 · reddit · 2026-10-05
A developer forked the Strata inference engine and, iterating heavily with Claude Code, tuned it for an IBM AC922 (dual POWER9 + 4x V100 SXM2, NVLink unified memory) running an 8B Qwen Q4KXL model. Prefill jumped from llama.cpp's 130 tok/s to a peak 7,357 tok/s (still 7,090 at a 252K-token prompt), decode 113 tok/s via MTP speculative decoding, with all 72 GiB of expert weights page-locked in RAM and GPUs pulling 70 GB/s over NVLink. Open-sourced as Strata-AC922.
More from Infra
- Ex-Waymo exec Abe Murray: CUDA and ISAs won't stay moats, huge chip innovation incoming — IgorCarron · 2026-11-20
- MIT and NVIDIA's QDA uses agents to auto-design LLM quantization recipes, beating top public ones by up to 2.2 points — omarsar0 · 2026-11-20
- Caixin: US remote-access cutoff and Nvidia CEO exit leave China's tech giants halting overseas compute deals — Gregory_C_Allen · 2026-11-20
- QuixiInfer: Rust Engine Runs 125B MoE Model on a Single RTX 5090 — QuixiAI · 2026-11-20
- QuixiInfer: from-scratch Rust engine runs DeepSeek with 256k context on a single RTX 5090 at ~30 tok/s — QuixiAI · 2026-11-20
- AMD's Lisa Su: data center to hit ~80% of revenue by 2027, sees $220B server CPU market — AccBalanced · 2026-11-20