230 agent-run experiments on LLM inference: 8 lessons from DeepSeek 4.1 Flash tuning
jiayq · x · 2026-10-29
Based on first-hand optimization of DeepSeek 4.1 Flash (2.41 ms per step, up to 2300 tok/s generation, 1800 tok/s on agent tasks), the author shares 8 lessons:
- Over 50% of gains came from beyond kernels: configs, graphs, launch order and system-level strategies; kernel work is a beaten path.
- MegaKernel might be a lie: on NVIDIA GPUs, gluing small ops into existing kernels or fusing small kernels into medium ones won biggest gains; vLLM's good TPU megakernel results suggest it's hardware-specific.
- KernelBench ≠ real serving: kernels positive in isolation often turn negative end-to-end; no simulator accurately models mixed compute/memory-bound latency.
- Verification is essential: with numerical tests and end-to-end quality evals, agents ran 230 proposals (94 verified useful) with humans only adding GPUs, avoiding token limits, and raising credit limits.
- Convergence speed is key and compute spend is real; at low concurrency SASS edits matter little — memory waits dominate.
- Next big gains lie in the distributed serving stack: telemetry and production-safe A/B testing, akin to classic recsys practice.
More from coding & agent
- Windows devs can format drives with ReFS to get ZFS-style fast cloning for worktrees — pvncher · 2026-10-29
- NVIDIA Launches CUDA Live: Introduction to CUDA Rust Programming — NVIDIAAI · 2026-10-29
- Developer trains custom game-playing models with his agent, builds Victris Arena — tristanbob · 2026-10-29
- Interface Lineage opens: 15 years of collected interfaces, 500+ patterns free to read — round · 2026-10-29
- Designer runs a daily bot to scout the web for design 'specimens' matching her taste — round · 2026-10-29
- Jason Kneen shares slop_eisley 0.6.22 roadmap for his AI-built FPS game — jasonkneen · 2026-10-29