Running an unquantized 61GB BF16 30B model on a 12GB RTX 4070 SUPER at 0.8 tok/s
Ill-Organization-38 · reddit · 2026-11-22
A developer's experimental local inference setup, Shark Cove, runs a 30B model with original unquantized BF16 weights (61GB) on an RTX 4070 SUPER (12GB VRAM) + 16GB DDR4, fully local, improving speed from 0.5 to 0.8–0.83 tokens/s (60–66%). The demo shows all 48/48 layers verified in full-depth inference, 32.96% lossless expert-routed reduction, and an SSD+CUDA staging scheme. It also includes hardware monitoring, memory management, and a Dynamic Token Window that resets active context budget each turn while storing conversation info separately. The goal is exploring what's possible on limited consumer hardware without sacrificing original weights, not beating cloud speeds; methods remain proprietary and it's proof-of-concept stage.
More from Infra
- Lenovo's enterprise AI infra revenue jumps 200%, outpacing cloud giants, $80B pipeline — shashib · 2026-11-22
- Dev rants against GPU-stacking: DeepSeek's <1GB RAM per 1M context shows efficiency wins — gnukeith · 2026-11-22
- Wiretok: open-source local monitor for Antigravity compaction headroom and API spend — phosphorhouse · 2026-11-22
- Musk details Tesla's in-car ~1B diffusion world model running at 36 fps in production — flngr · 2026-11-22
- Nvidia plunges 14% after Reuters reports SEC probe into its accounting practices — mark_k · 2026-11-22
- ING fails to sell $500M data center loan bundle, forced to sell loans one by one — churchkey · 2026-11-22