Running GPT-OSS 120B on a 4070 Ti at 21 tok/s via Aggressive MoE Caching
JayB_Official · reddit · 2026-08-09
A developer successfully ran the 59GB GPT-OSS 120B model locally on a 12GB VRAM RTX 4070 Ti, achieving 21 tok/s generation speeds.
Key Implementation Details:
- Leveraging MoE Architecture: Kept static parameters in RAM, hot experts on the GPU, and cold experts on an NVMe SSD.
- Dynamic Caching: Streamed SSD experts into fixed GPU buffers on-demand, adaptively caching frequently used experts in persistent GPU memory.
- Aggressive Approximation: Hit 21.16 tok/s using 16 adaptive GPU hot expert slots and a top-1 approximation mode.
The author demonstrated that with 'caveman engineering' using open-source tools, consumer hardware can run ultra-large models.
More from coding & agent
- Constraints you force on models today will become tech debt in months, warns dev — zack_overflow · 2026-10-16
- Liquid AI tutorial: fine-tune your d1-3B decision model in 30 minutes on a free GPU — helloiamleonie · 2026-10-16
- Alibaba open-sources OpenGCUA, a full-stack generalist computer-use agent; +19.6 points on OSWorld-v2 across 8 models — Tongyi-MAI · 2026-10-16
- BotBeat: a social network where all users are AI agents, exposed via 32 MCP tools — Dismal_Piccolo4973 · 2026-10-16
- DKGI paper: agents write their own tools to search knowledge graphs, beating baselines by 6.7 Hit@1 — yuz9yuz · 2026-10-16
- Dev open-sources lgtmaybe: running PR code reviews on a local 27B model via hunk splitting — mattjcoles · 2026-10-16