Running 284B Model on Single 96GB GPU: Speculative Decoding & VRAM Allocation Test
FantasticNature7590 · reddit · 2026-08-13
The author conducted in-depth inference optimization tests on a single RTX PRO 6000 (96GB) for the DeepSeek V4 Flash 284B model, exploring best practices for limited VRAM.
- DSpark speculative decoding: Matching VRAM usage, DSpark speculative decoding improved generation speed by 15-17%.
- Counter-intuitive VRAM allocation: Moving the DSpark drafter to system RAM freed VRAM for more target model layers. Since target layers are read more frequently, this actually yielded a 4.4% speed increase.
- Speculative token optimization: Generating 3 speculative tokens was the sweet spot; while 2 had higher acceptance, 3 produced more useful accepted tokens per verification round.
- q80 KV cache is virtually lossless: Using q80 quantized KV cache expanded context from 256K to 768K with minimal decode speed loss, retaining retrieval capabilities at 900K tokens.
The setup uses Ryzen 9 9950X + 96GB DDR5, splitting the model (21 layers on GPU, 19 in RAM), making it memory-bandwidth bound.
More from Infra
- Geek Test: Running a 122B Parameter LLM on an Obsolete Laptop — _TheGreatDreamer_ · 2026-08-13
- Cross-Provider Speculative Decoding: Acceptance Rate Collapses Past 32K Context — hoyasgirl25 · 2026-08-13
- Why Space is a Terrible Place to Cool an AI Data Center — CackleRooster · 2026-08-13
- Texas to hit 10GW power as nuclear becomes terrestrial compute fuel — NinaDSchick · 2026-08-13
- Why Space is a Terrible Place to Cool a Data Center — CrankyBear · 2026-08-13
- CoreWeave Warns Investors of Time and Cost to Shift Away from Nvidia Chips — dinabass · 2026-08-13