Tokenomos open-sourced: rewritten vLLM frontend makes small-model inference ~3x faster on RTX 5090
bdsqlsz · x · 2026-11-01
KBlueleaf open-sourced Tokenomos, a local inference engine for small VLMs that speeds up serving not via GPU kernels or quantization tricks—the model code is borrowed wholesale from vLLM—but by fully rewriting the frontend and moving scheduling, KV cache/state management, sampling and stop-checking onto the device, so the CPU is never the bottleneck.
Benchmarks on RTX 5090:
- LFM3-VL 2B hits 79,030 decode tok/s at batch 512 vs 22,448 for vLLM-based Kohaku (3.5x)
- Gemma 4 E4B reaches 29,524 tok/s vs vLLM's 17,213
- GPU idle per step at 512 sequences drops to 0.23 ms vs vLLM's 14.37 ms
- Host cost per decode step stays nearly flat as sequences grow (0.12→0.41 ms)
Supports LFM3, Qwen3.5-3.8, Gemma 4; Apache-2.0 licensed.
More from Infra
- Meta's $97.5B mega raise: $25B share sale 5x oversubscribed, Nvidia commits $3B, AMD $1B — Beth_Kindig · 2026-11-01
- Server CPU shipments to grow 31% in 2026, over 40% in 2027 on inference and agentic AI demand — Beth_Kindig · 2026-11-01
- "Family GPU" box shares compute over WiFi, runs 750B model at ~8 tok/s — takuonline · 2026-11-01
- Interactive visualizer lets you zoom into Esperanto's ET-SoC-1 AI chip, down to FMA instructions — yaroslavvb · 2026-11-01
- Musk: Colossus 2 to host ~1.2M Blackwell GPUs by end of year — scaling01 · 2026-11-01
- Local Maximus meme: six RTX 3090s, 256GB RAM, servant to no API — gklambauer · 2026-11-01