80B MoE Model Runs in a Chrome Tab: llama.cpp WebGPU Backed Pages Experts From Disk
naklitechie · reddit · 2026-10-13
The author ported last week's hand-written disk-tier streaming into llama.cpp's official WebGPU backend, so any MoE model llama.cpp supports can now run browser-side with no per-model engine. Real recording on MacBook Pro M4 Pro (24GB): Qwen3-Next 80B-A3B (48.4GB file, only 4.89GB resident on GPU) loads in 48s and decodes at 2.9 tok/s as proof of concept; OLMoE 1B-7B hits 22.4 tok/s; LFM2 8B-A1B 17.1 tok/s; Qwen3-30B-A3B (14.7GB) also runs.
How: the GGUF downloads once into OPFS; GPU holds dense layers, KV cache and a fixed expert-slot budget per layer; at each MoE layer the router's picks are read back and non-resident experts are streamed from OPFS into LRU slots via sync access handles in workers. Changes: a new buffer type, slot-mapped MULMATID shaders, residency logic.
Correctness: paged vs unpaged replies identical (9/9); vs llama.cpp on Metal, 4-8 of 9 match — misses are WebGPU-vs-Metal rounding at near-ties; against the CPU backend WebGPU is as close as Metal.
Limits: 359MB disk reads per token on the 80B, slow prompt processing, Chrome/Edge 137+, Apple silicon only. Fun bit: asked how it could run in a browser tab, the 80B itself answered "It can't." It was running in a browser tab.
More from Infra
- Cache-busting bug masked a 24x GPU gap: 11 confident falsehoods, none from the model — colinmcnamara · 2026-10-13
- Developers weigh running sandboxes on a Mac mini via apple/container to halve Modal bills — TheZachMueller · 2026-10-13
- How One Company Serves 40 Staff Local AI on 4x RTX 6000 Pro With vLLM and n8n — q-admin007 · 2026-10-13
- AMD Helios racks estimated at $7.33M, nearly matching Nvidia Vera Rubin's $7.52M — Beth_Kindig · 2026-10-13
- Slash Your Modal Sandbox Bill With a Mac Mini? Devs Weigh Cost vs Privacy — andersonbcdefg · 2026-10-13
- NetBird's Proxmox one-liner hits 4,093 installs in 30 days on Community Scripts — victorialslocum · 2026-10-13