Transformers Can Now Run Natively in vLLM
huggingface · x · 2026-07-14
Hugging Face announced that Transformers models can now run at native speeds in vLLM, with throughput often matching or exceeding hand-written implementations.
The key shift: previously, a new architecture required maintaining two separate implementations—one in Transformers for training/research and another in vLLM for high-performance inference. Now, model authors only need to implement it once in Transformers to leverage vLLM's optimized inference stack.
Benchmarks indicate that this backend maintains throughput comparable to or higher than native vLLM across models ranging from 4B to 235B parameters, covering both tensor parallelism and MoE scenarios. The authors emphasize that a single, readable model implementation can now serve training, fine-tuning, evaluation, RL rollout, and production inference.
More from Infra
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- xAI's Colossus 1 hits ~14.6 exaflops FP64, ~5x Top500 leader — cis_female · 2026-10-27
- Lambda runs 19 B200 nodes in a 16-node power budget, boosting throughput 24% — TheZachMueller · 2026-10-27
- Podcast: Micron's HBM roadmap, Meta's AI infra, and NVIDIA's supply chain lock-in — BenBajarin · 2026-10-27
- SGLang MegaKernel + PD Pushes the TCO-Latency Frontier on GB300, SemiAnalysis Benchmarks Show — AccBalanced · 2026-10-27
- Scaling self-hosted Convex: compiling TypeScript Convex functions to WASM — leventov · 2026-10-27