Burkov: Jev can't beat LLM speed-quality tradeoff — here's the KV-cache trick that matches it
burkov · x · 2026-09-23
- Andriy Burkov argues Jev can't be both faster-to-decision and as good as an LLM: time-to-first-token equals compute spent deciding, so being faster means losing quality.
- He shows how to get identical classification speed from a plain LLM: build parallel requests sharing a common prefix in KV cache, send only the option word, read a single 0-9 score token per option, then softmax the scores.
- Without thinking this costs only prefix embedding + one KV-cache append + one digit generation; with thinking enabled quality far exceeds Jev.
Related event: Burkov Dismisses Mystery Project Jev as Likely BS(5 posts)→
More from Infra
- Netlify ships Server: a plain-language map of serverless functions vs long-lived servers vs VMs — thisiskp_ · 2026-11-12
- Show Reddit: ge, a Rust GGUF engine, benchmarks against llama.cpp on a 4GB RTX 3050 — Raptor2496 · 2026-11-12
- Musk: 100GW of space solar equals ~800GW on Earth, and ground solar needs batteries — elonmusk · 2026-11-12
- Real-world test: Kimi K3.1 Pro costs 5.4x less than Opus 5.5 per task despite 2x list-price gap — Arindam_1729 · 2026-11-12
- Dino10B: first 10-billion-scale nearest-neighbor search benchmark, 10.3TB of DINOv3 embeddings — philipvollet · 2026-11-12
- Many announced data centers may never get built, TechCrunch argues — TechCrunch AI · 2026-11-12