8B MoE hits 7.2 tok/s on a TV box with 1GB RAM, experts paged from flash
petrusenko_max · x · 2026-11-13
A developer reports running an 8B mixture-of-experts model at 7.2 tok/s on a TV box with only 1GB of usable RAM, paging experts from flash storage on demand. A dense 9B model on the same box crawls at 0.27 tok/s — about 26x slower.
The takeaway: for on-device agents, active parameters per token decide what ships, not RAM headroom. The work is part of the open-source smolmaxxing project, which pushes running LLMs on the smallest devices possible.
More from Infra
- Microsoft data center outage sparks jokes about servers 'going on protest' — SuB8u · 2026-11-13
- Notion indexes 10B+ docs on turbopuffer, cuts search costs 80% — DevinLewtan · 2026-11-13
- Even cutting all China GPU purchases leaves massive unmet compute demand — peterjliu · 2026-11-13
- Snowflake and Databricks are converging into the same platform, feature by feature — Zachly · 2026-11-13
- Inference chip startup Fractile raises $720M Series C six months after its $220M Series B — AjdDavison · 2026-11-13
- Another Maine town extends data center moratorium as local opposition spreads to 12+ communities — jonippolito · 2026-11-13