Transformers Can Now Run Natively in vLLM
huggingface · x · 2026-07-14
Hugging Face announced that Transformers models can now run at native speeds in vLLM, with throughput often matching or exceeding hand-written implementations.
The key shift: previously, a new architecture required maintaining two separate implementations—one in Transformers for training/research and another in vLLM for high-performance inference. Now, model authors only need to implement it once in Transformers to leverage vLLM's optimized inference stack.
Benchmarks indicate that this backend maintains throughput comparable to or higher than native vLLM across models ranging from 4B to 235B parameters, covering both tensor parallelism and MoE scenarios. The authors emphasize that a single, readable model implementation can now serve training, fine-tuning, evaluation, RL rollout, and production inference.
More from Infra
- Nebius says SlimSpec speeds speculative decoding 8–9% without shrinking the vocabulary — Arindam_1729 · 2026-07-21
- NVIDIA brings its Cosmos 3 Edge world model to Jetson for on-device robot control — liu_mingyu · 2026-07-21
- A silicon photonic reservoir chip compensates fiber distortion in real time at 28 Gbps — bravo_abad · 2026-07-21
- Chamath says open-sourcing Grok would push AI margins from models to infra and apps — Dan_Jeffries1 · 2026-07-21
- EU AI competitiveness is under pressure as firms double down on chips, ethics, and talent — nordicinst · 2026-07-21
- AI bottlenecks are shifting to memory, optics, yield control and power — thedealdirector · 2026-07-21