French team drops vLLM/PyTorch for Rust+CUDA and FP4/FP8, huge single-GPU inference gains
IgorCarron · x · 2026-11-18
As the French team speyronnet nears launch of their consumer inference engine ibou, they shared an aggressive optimization push for their in-house short-synthesis model Beaupré: replacing vLLM and PyTorch entirely with Rust + CUDA.
The optimization stack:
- Full rewrite of the inference stack in Rust + CUDA;
- Mixed-precision inference: part FP4, part FP8 — the right precision in the right place;
- A drafter carefully trained on their target workload for speculative decoding;
- Plenty of undisclosed small hacks (the "secret sauce").
The result is a dramatic performance improvement on a single GPU — a textbook case of bypassing general inference frameworks for a hand-rolled extreme-performance stack.
More from Infra
- Bloomberg: New US Chip Rule Freezes $480M Nvidia GPU Loan for ByteDance in Malaysia — rohanpaul_ai · 2026-11-18
- AI raises the cost of weak cloud architecture, says resilience expert — DavidLinthicum · 2026-11-18
- Harvard paper: 94.8% of coding agent tokens are cached re-reads, 25x energy gap — rohanpaul_ai · 2026-11-18
- LlamAmpere v0.7: Qwen 27B at Full Context on 12GB Cards, 70 TPS — Brief-Tap-6616 · 2026-11-18
- Qwen4-Max lands on DeepInfra with 1M context at $1.413/M input tokens — gharik · 2026-11-18
- TensorScale Visualizes Every Inference Stage and Benchmarks Against a Leading Hosting Provider — TerryTangYuan · 2026-11-18