500K Embedding Tokens/Sec on One GPU: Superlinked's Small-Model Serving Stack
AI Engineer · youtube · 2026-09-20
Daniel Svonava of Superlinked, speaking at AI Engineer, on the infrastructure for serving small models.
Core claims
- Embeddings are the "no-brainer entry point": a single mid-range GPU turns half a million tokens per second into embeddings in the low tens of milliseconds, while a managed endpoint costs orders of magnitude more and takes hundreds of milliseconds.
- Small models are catching the frontier: a small model fits on one GPU two or three generations old and, for a specific task, is now at or beyond the frontier — the frontier is flattening while small open models climb.
- Don't prompt one 27B model for everything: slice the workload into tasks and pick the model trained for each, so a contract-review agent ends up running nine models. The model that has seen the most Vietnamese receipts wins Vietnamese receipt OCR.
Three bottlenecks
- Open-source serving tools ship untuned, so adopting one becomes a research project.
- Top-down routers, built to spread one big model across GPUs, choke on many small requests because their view of worker state is always stale; utilization stalls near 30%.
- LoRAs and overnight fine-tunes turn every deployment into a conversation between AI and infrastructure engineers, killing velocity.
Superlinked's answer
- Open source under Apache 2.0 from control plane to GPU.
- A gateway annotates a request without fully parsing it and drops it into a shared queue; workers pull and form their own batches — that inversion doubled cluster throughput.
- A Rust sidecar abstracts 50 adapters over three runtimes, and an autoresearch loop ships every model already tuned.
- One result: an 80-cent LoRA lifted retrieval on German legal text by 18%.
More from Infra
- NEAR AI Brings Confidential Inference to Bittensor Subnet SayGm, an OpenRouter-Style Router — markjeffrey · 2026-09-20
- Game engines and inference engines both boil down to multi-user batching, and agentic bots — yunta_tsai · 2026-09-20
- Tutorial: speculative decoding in vLLM to cut LLM latency and double tokens per second — MaiaStudios · 2026-09-20
- Charles Frye: KV Compression Fails Rarely but Expensively at Long Context, and New Models Need It Less — charles_irl · 2026-09-20
- Two vLLM bugs hid in plain sight: Mamba state cache, decode-before-prefill and a 32-bit wrap — AI Engineer · 2026-09-20
- Analyst: enterprises overpay 10-20x for cloud AI as on-prem inflection nears — DavidLinthicum · 2026-09-20