Nebius Engineers Break Down How to Serve Open LLMs Fast and Cheap in Production

AI Engineer · youtube · 2026-10-04

At AI Engineer World's Fair 2026, Nebius' Dylan Bristot and Sujee Maniyam walk through what it takes to run open LLMs in production as they near parity with proprietary models.

Key points:

Closed APIs vs self-hosting: open models are now competitive, cutting cost and lock-in; the talk frames the full inference → data → post-training → deployment loop.
Hardware: NVFP4 quantization on latest NVIDIA hardware.
Engine selection and cache-aware routing, which beats naive load balancing for LLM serving.
Speculative decoding, including training custom draft models on your own traffic.
KV cache offloading and disaggregated prefill/decode.
Finding the quantization sweet spot between accuracy and throughput.

A layer-by-layer roadmap (hardware → engine → routing → decoding → caching) for teams self-hosting open models, with Nebius Token Factory docs linked.

Original post →

Nebius Engineers Break Down How to Serve Open LLMs Fast and Cheap in Production

More from Infra