Transformers Can Now Run Natively in vLLM
huggingface · x · 2026-07-14
Hugging Face announced that Transformers models can now run at native speeds in vLLM, with throughput often matching or exceeding hand-written implementations.
The key shift: previously, a new architecture required maintaining two separate implementations—one in Transformers for training/research and another in vLLM for high-performance inference. Now, model authors only need to implement it once in Transformers to leverage vLLM's optimized inference stack.
Benchmarks indicate that this backend maintains throughput comparable to or higher than native vLLM across models ranging from 4B to 235B parameters, covering both tensor parallelism and MoE scenarios. The authors emphasize that a single, readable model implementation can now serve training, fine-tuning, evaluation, RL rollout, and production inference.
More from Infra
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
- Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11