Reddit Thread Argues One-Off Inference Engines Will Become the Norm, Ousting vLLM/llama.cpp
netherreddit · reddit · 2026-09-30
The author argues that 'overfit' one-off inference engines targeting single model/hardware combos will become the norm, leaving general engines like llama.cpp and vLLM too slow for most users.
The argument rests on three axioms:
- Smaller, less general codebases iterate faster;
- AI coding is dropping the barrier to building an engine;
- 'Make tok/s go up' on a single fork is a fully specified task — ideal for fully autonomous AI implementation with no human bottleneck.
Implications: general engines will perpetually lag in dev speed and token speed, while one-offs (ninfer, dwarfstar, Splash, llamAmpere, gufo, etc.) keep appearing and dying within months. Elements like API/CLI/GUI/benchmarking/model format may standardize — a possible open-source project opportunity; 'half-general' engines targeting one hardware platform may emerge; hardware-specific communities will form.
Alternative futures: general engines could survive by plugin-izing model/hardware-specific kernels (ship an .inferencerecipe alongside the .gguf) or by AI-ifying their own dev workflow.
More from Infra
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Stratechery's Fictional Obituary: Imagining Jensen Huang's 2026 Exit From Nvidia — Gregory_C_Allen · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16
- Colossus 2's 1.2M B200s could generate ~$400B/year in tokens at DeepSeek Flash rates — bookwormengr · 2026-11-16
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- Qdrant 1.20.1: 50% fewer cold-storage reads, ~2x faster quantization, agent-ready setup — qdrant_engine · 2026-11-16