Custom LLM Inference Engine: 10-20% Faster than vLLM
AlpinDale · x · 2026-08-20
The author shares a from-scratch LLM inference engine built over the past 2+ years, featuring a custom MLIR-based compiler.
- Performance: Currently 10-20% faster than vLLM on most GPUs, with expectations for further gains after compiler work.
- Features: Currently barebones, supporting only Qwen3/3.5, FP8/NVFP4 quantization, KV cache quantization, and dflash.
- Goal: Aiming to release this year and seeking community feedback on desired features (e.g., KV offloading, hybrid CPU-GPU inference) compared to vLLM/SGLang/llama.cpp.
More from Infra
- Animation puts AI data center water usage in context amid debate — ATTlKA · 2026-08-20
- Local LLM quantization guide: Hardware thresholds for FP8, NVFP4, and more — Ill_Dragonfruit_3547 · 2026-08-20
- Mojo integrated with MLIR stack, running matrix multiplication on Corsair in days — clattner_llvm · 2026-08-20
- Blueprint raises $1M+ pre-seed led by a16z to speed up hardware iteration — Scobleizer · 2026-08-20
- Mac can now run a 27B model locally that codes, reasons, and sees — TheMoonMidas · 2026-08-20
- RTX PRO 6000 Blackwell Max-Q Review: Ideal for Multi-GPU Towers — TheZachMueller · 2026-08-20