Running 284B Model on Single 96GB GPU: Speculative Decoding & VRAM Allocation Test
FantasticNature7590 · reddit · 2026-08-13
The author conducted in-depth inference optimization tests on a single RTX PRO 6000 (96GB) for the DeepSeek V4 Flash 284B model, exploring best practices for limited VRAM.
- DSpark speculative decoding: Matching VRAM usage, DSpark speculative decoding improved generation speed by 15-17%.
- Counter-intuitive VRAM allocation: Moving the DSpark drafter to system RAM freed VRAM for more target model layers. Since target layers are read more frequently, this actually yielded a 4.4% speed increase.
- Speculative token optimization: Generating 3 speculative tokens was the sweet spot; while 2 had higher acceptance, 3 produced more useful accepted tokens per verification round.
- q80 KV cache is virtually lossless: Using q80 quantized KV cache expanded context from 256K to 768K with minimal decode speed loss, retaining retrieval capabilities at 900K tokens.
The setup uses Ryzen 9 9950X + 96GB DDR5, splitting the model (21 layers on GPU, 19 in RAM), making it memory-bandwidth bound.
More from Infra
- Dev publicly asks neoclouds for discounts on a $30k GPU compute purchase — Sauers_ · 2026-10-16
- New hardware day: local LLM deployment the moment it arrives — lxfater · 2026-10-16
- Together AI's score centering fixes RL training collapse under quantized inference — nrehiew_ · 2026-10-16
- Inference is splitting prefill and decode across specialized chips, not just H100s — AccBalanced · 2026-10-16
- Rebasely launches in beta: Git hosting built for humans and coding agents — handotdev · 2026-10-16
- Drama3 TTS lands in audio.cpp: up to 5x real-time on RTX 5090, <8GB VRAM mode too — Acceptable-Cycle4645 · 2026-10-16