Running an unquantized 61GB BF16 30B model on a 12GB RTX 4070 SUPER at 0.8 tok/s

Ill-Organization-38 · reddit · 2026-11-22

A developer's experimental local inference setup, Shark Cove, runs a 30B model with original unquantized BF16 weights (61GB) on an RTX 4070 SUPER (12GB VRAM) + 16GB DDR4, fully local, improving speed from 0.5 to 0.8–0.83 tokens/s (60–66%). The demo shows all 48/48 layers verified in full-depth inference, 32.96% lossless expert-routed reduction, and an SSD+CUDA staging scheme. It also includes hardware monitoring, memory management, and a Dynamic Token Window that resets active context budget each turn while storing conversation info separately. The goal is exploring what's possible on limited consumer hardware without sacrificing original weights, not beating cloud speeds; methods remain proprietary and it's proof-of-concept stage.

Original post →

More from Infra

Infra channel →