FriendliAI: agents change inference economics — open-weight run 5.5x cheaper, one customer 7x faster
AI Engineer · youtube · 2026-09-20
AI Engineer interviews Byung-Gon Chun, whose team invented continuous batching and inspired a widely used open-source serving framework.
Key points:
- The same coding agent building a tower defense game finished at a usable level on an open-weight model, roughly 5.5x cheaper than a closed frontier model.
- Agentic inference differs from chat: the unit is the task (plan/act/observe loops running minutes to hours, parallel sub-agents, ever-growing context), not the request.
- Internal traces show consecutive steps share enormous prefixes; recomputing them is wasted compute.
- FriendliAI rebuilt its stack around task completion: prefix caching, hierarchical KV cache across GPU/host memory/disk, cache-aware routing, and agent-aware scheduling.
- One customer's split test showed 7x faster completion with a lower error rate.
Full timestamps included in the video.
More from Infra
- Ben Bajarin: Agentic AI will spawn an 'agentic native' CPU tier in datacenters — BenBajarin · 2026-09-20
- Qwen 3.8 Next Flash at 3.05bpw EXL3 runs like Q8 on 3x RTX 3090s, dev reports — nicholas_the_furious · 2026-09-20
- Redditor pleads with FP4 inference engine builders: small dense models at FP4 are cooked — buttplugs4life4me · 2026-09-20
- Chips will depreciate more slowly as materials change, and thermal compute is still coming — beffjezos · 2026-09-20
- Devs debate running stateful AI agent runtimes on Cloudflare Workers and other edge runtimes — merlinofthewater · 2026-09-20
- Questioning Jev's Speed Pitch: Why Route Through OpenRouter and Add Latency? — deliprao · 2026-09-20