FriendliAI: agents change inference economics — open-weight run 5.5x cheaper, one customer 7x faster
AI Engineer · youtube · 2026-09-20
AI Engineer interviews Byung-Gon Chun, whose team invented continuous batching and inspired a widely used open-source serving framework.
Key points:
- The same coding agent building a tower defense game finished at a usable level on an open-weight model, roughly 5.5x cheaper than a closed frontier model.
- Agentic inference differs from chat: the unit is the task (plan/act/observe loops running minutes to hours, parallel sub-agents, ever-growing context), not the request.
- Internal traces show consecutive steps share enormous prefixes; recomputing them is wasted compute.
- FriendliAI rebuilt its stack around task completion: prefix caching, hierarchical KV cache across GPU/host memory/disk, cache-aware routing, and agent-aware scheduling.
- One customer's split test showed 7x faster completion with a lower error rate.
Full timestamps included in the video.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04