Running GPT-OSS 120B on a 4070 Ti at 21 tok/s via Aggressive MoE Caching
JayB_Official · reddit · 2026-08-09
A developer successfully ran the 59GB GPT-OSS 120B model locally on a 12GB VRAM RTX 4070 Ti, achieving 21 tok/s generation speeds.
Key Implementation Details:
- Leveraging MoE Architecture: Kept static parameters in RAM, hot experts on the GPU, and cold experts on an NVMe SSD.
- Dynamic Caching: Streamed SSD experts into fixed GPU buffers on-demand, adaptively caching frequently used experts in persistent GPU memory.
- Aggressive Approximation: Hit 21.16 tok/s using 16 adaptive GPU hot expert slots and a top-1 approximation mode.
The author demonstrated that with 'caveman engineering' using open-source tools, consumer hardware can run ultra-large models.
More from coding & agent
- bindureddy Teases Open-Source AI Coding Harness with Free Model Support — bindureddy · 2026-08-09
- 15 Crucial AI Agent Design Patterns: From Single to Multi-Agent Orchestration — MaryamMiradi · 2026-08-09
- swyx Hosts 'Kill My SaaS' Hackathon with $10k Prize, Over 600 Applicants — GregKamradt · 2026-08-09
- freephdlabor: Open-Source Multi-Agent System for End-to-End Scientific Research — tom_doerr · 2026-08-09
- A Comprehensive Guide to LLM Inference Optimization and Deployment — abhijithneil · 2026-08-09
- Developer Tests Codex Multi-Thread Agent Coordination: Fascinating Yet Scary — KarelDoostrlnck · 2026-08-09