Distilling DeepSeek V4 Flash to a 4B model on DGX Spark: 26 hours, 22ms per judgment
Dan_Jeffries1 · x · 2026-09-19
Developer @taroleo shares a local distillation experiment: spending 26 hours on a DGX Spark to distill the judgment capability of the 157GB-weight DeepSeek V4 Flash into a 4B model.
Key takeaways:
- Dropping text/ generation and specializing purely in judgments pushes local inference down to 40ms
- The 4B student, about 1/20 the teacher's size, outperforms the teacher's instant-response mode
- Each judgment takes 22ms, fully on-device
The insight: instead of having a big model emit full JSON, strip generation down to classification-only and gain order-of-magnitude speed. A useful reference for distillation + edge deployment.
More from Infra
- $250 of modded mining cards, 30GB VRAM: old i7 PC runs Qwen at 30 tok/s with patched drivers — HFq_Dev · 2026-09-19
- Reading a Pretraining Run: A P0/P1/P2 Metric System for Monitoring LLM Pretraining — SonglinYang4 · 2026-09-19
- Dual RTX 5060 Ti Only Gets 10 t/s on Qwen3.8-Flash-Next, Seeking Config Advice — MkGod · 2026-09-19
- Hacking open-source model behavior with sglang's scoring endpoint, no fine-tuning needed — BLUECOW009 · 2026-09-19
- Positron raises $875M at $5B valuation as its co-founder calls anti-data-center talk a "Chinese psyop" — 20VC · 2026-09-19
- Dual RTX 3060 Local LLM Setup: 100k Context at 600 tok/s, $2k Upgrade Paths — gnoremepls · 2026-09-19