Revolut cut first-token latency from 10s to 1.6s on the same 56 GPUs with 4-bit weights and 256K context cap
ccerrato147 · x · 2026-11-05
A Revolut inference optimization case: on the same 56 GPUs, first-token latency went from over 10 seconds in April to 1.6 by August while handling more than twice the traffic. Tactics included capping context at 256K tokens, running weights at 4 bits, and tuning for cache hit rates above 90%. The author adds an adoption insight: usage caps actually save tokens—rationed engineers save tokens for the hardest tasks, while unrationed ones hand agents everything (code, tickets, inbox), making rationing the real cost.
More from Companies & People
- Mistral AI's Office Hours Return Nov 5: Studio and Vibe with Team Guest — MistralAI · 2026-11-05
- Zach Lloyd: Don't vibe code your software factory — prompt-chained agents don't scale — vikvang1 · 2026-11-05
- Hands-on workshop on safely building multi-agent systems at Slush Helsinki, Nov 19 — brada · 2026-11-05
- Anima Anandkumar to keynote SC26 supercomputing conference in Chicago — AnimaAnandkumar · 2026-11-05
- 180 builders to hack consumer mobile apps at YC office; winners get YC interview — seatedro · 2026-11-05
- SOTAwives podcast hosts ex-Anthropic engineer right after his last day — ycombinator · 2026-11-05