ZSE inference engine: 30x faster cold start than vLLM, no PyTorch needed
tom_doerr · x · 2026-08-14
ZSE is a zero-dependency LLM inference engine that eliminates PyTorch, Triton, bitsandbytes, and transformers, using pure Python, ctypes, and a kernel compiler to emit CUDA, ROCm, and Metal directly. Verified on Modal (T4, L4, A10G, A100), DigitalOcean (MI300X), and Apple M1, ZSE INT4 achieves dramatically faster cold starts compared to vLLM AWQ: for Qwen2.5-7B on T4, 7.25s vs 218.96s (30.2x speedup), and on L4, 5.58s vs 145.22s (26x). Models load in seconds with minimal memory footprint.
More from Infra
- OpenAI Releases GPT-5.6 Builder Guide: Slash Agent Bills from $33 to $1.33 — xiaohu · 2026-08-14
- Google Releases Free Masterclass on GPUs — mdancho84 · 2026-08-14
- Stable ComfyUI on AMD Linux: Docker image pins ROCm runtime — zychu- · 2026-08-14
- Micron Trades at 9x Forward P/E, Sparking Valuation Debate — JOBhakdi · 2026-08-14
- Dual MI50 32GB Build Advice: Setting Up Hermes and vLLM — opoot_ · 2026-08-14
- Why Kubernetes CPU Limits Are Harmful: A Deep Dive into Performance Pitfalls — iljanevo · 2026-08-14