Strata runs a 125B model on a 12GB GPU at 96 tok/s, one-click local install
daluoseo · x · 2026-10-03
A look at Strata, an open-source inference engine (7.2k GitHub stars) that runs Qwen3.8-Flash-Next — a 125B-parameter model that normally needs a server — on consumer gaming PCs with just 12GB of VRAM (NVIDIA or AMD), reportedly hitting 96 tokens/s.
Highlights:
- One-click install for Windows/Linux, optional Docker
- Local OpenAI/Anthropic-compatible API
- Chat, code, image input, 128K context (IQ3S quant, tested on RTX 5070)
- Free and open source, built on ggml
Caveat: the "Qwen3.8-Flash-Next" model name doesn't match official Qwen releases, so the project's claims warrant verification.
Related event: Strata Lets Consumer PCs Run 180B Models Locally(3 posts)→
More from Infra
- Gridline launches one-API fine-tuning, distillation and batch inference for open models — Scobleizer · 2026-11-19
- Ascend output could hit ~20M chips next year; memory and packaging, not logic, are the bottleneck — teortaxesTex · 2026-11-19
- AMD Loom boosts Qwen3.5-4B prefill up to 2.6x on Radeon 890M laptops — jfowers_amd · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS Lambda Adds GPU Support via Managed Instances for Inference and Media Workloads — DavidWells · 2026-11-19
- Nvidia Allegedly Made GPUs Artificially Scarce to Keep Prices Sky-High — GaryMarcus · 2026-11-19