200 tok/s on 8GB VRAM: dev benchmarks 6 small models for local AI
TheMoonMidas · x · 2026-09-06
Developer @nettermina shows a max-optimized local setup: no $15,000 GPU needed — an old 8GB VRAM card (tested on a forgotten 3070 Ti laptop) runs Windows with 200 tok/s, 128k context, and text/image/audio support.
The team pulled every instruct model since May that fits 8-12GB and ran six through the same 20 tasks (JSON extraction, code, tables, arithmetic, deep-prompt fact recall, tool calls, photo reading): Gemma 4 E4B, Gemma 4 12B, Qwen3.5-9B, Qwen3.5-4B, Ornith-1.5-9B, and a Qwen3.8-9B distill.
Result: Gemma 4 E4B isn't the top scorer — Qwen3.5-9B and Ornith beat it on long context and fine print in photos — but E4B won the pick for what a newcomer on an old card actually needs. A reproducible recipe is included for anyone with a 5-year-old laptop.
More from Infra
- SurfSense: an open-source local NotebookLM with MCP agents and a prompt-caching lesson — FurtiveMirth · 2026-10-23
- Meta researcher: horrifying infra, not talent, is why OSS can't catch closed frontier — A_K_Nain · 2026-10-23
- Jane Street and NVIDIA speed up Mamba-2 with custom cuTile kernels — blelbach · 2026-10-23
- LatentLoop: asymmetric compute allocation cuts looped-MoE training cost by 19% — pmttyji · 2026-10-23
- Same AI request emits 8.5x more CO2 in Texas than in Sweden — jlundberg · 2026-10-23
- SK Hynix Pulls Plug on Solidigm US IPO as Chip Stocks Slide, Shares Down ~30% This Quarter — rohanpaul_ai · 2026-10-23