From-scratch Rust+CUDA inference engine for DeepSeek hits 1740 TPS on 4x B300
saranormous · x · 2026-11-14
A developer built an inference engine for deepseek v4.1-flash from scratch in pure Rust + CUDA — no PyTorch, no SGLang, no DSLs. It achieves 1740 TPS decode on 4x B300, 2060 TPS on 8x B300, and 12s prefill for 1M tokens. Code is open-sourced on GitHub; saranormous amplified it as a glimpse of open-source, continually-learning inference engines.
More from Infra
- Cloudflare open-sources Kitesurf, a stateless browser for AI agents built on Workers — ritakozlov · 2026-11-14
- Google's week: two 8th-gen TPUs, Gemini 4 Carbon & Hydrogen, Robotics 2.5 — sundarpichai · 2026-11-14
- Investor Reveals Dual NBIS-AKAM Position: Betting Both Ends of the Compute Stack — pdamodaran · 2026-11-14
- Nina Schick: the first computing cycle where demand isn't capped by humans — NinaDSchick · 2026-11-14
- MiniMax H3 Image's ~50GB stack runs on 8GB VRAM at 57s/image — how is there no OOM? — wreck_of_u · 2026-11-14
- AMD raises guidance: revenue set to grow 80%+ next year as data center AI revenue more than doubles — Beth_Kindig · 2026-11-14