Andriy Burkov on why Jev-style LLM calibration can't beat an LLM at real probabilities
burkov · x · 2026-09-23
ML author Andriy Burkov posted a technical critique of Jev:
- Calibration isn't ground truth: You can RL-finetune an LLM using the gap between softmax probabilities and "real" probabilities from training data (what people call calibration), but the model still won't learn to output true probabilities for arbitrary questions — ground-truth probabilities can't exist for every possible question, from "civilian or military?" to "buy or sell this stock?" to which sock to put on first.
- Speed implies a quality ceiling: An LLM's time-to-first-token reflects the compute spent on a decision. If Jev emits its full output faster than an equal-sized LLM prints its first token, it has automatically lost on quality.
His conclusion: such fast, small decision models gain speed precisely by giving up probabilistic reliability on open-ended questions.
Related event: Burkov Dismisses Mystery Project Jev as Likely BS(5 posts)→
More from Models
- a16z's Martin Casado: Google's Carbon is 'really good at code' — xennygrimmato_ · 2026-11-12
- Blogger vouches for rumored Gemini 4 Carbon: good taste, cost-effective — intellectronica · 2026-11-12
- Walden Robotics' Sage model quadruples robot uptime on Toyota factory lines — adnothing · 2026-11-12
- Gemini won't let you keep chat history while opting out of AI training — Google bundles both into one setting — arshdeep · 2026-11-12
- Gemini 4 Carbon Flex tier now costs the same per token as GPT 6.1 Terra — mariofilhoml · 2026-11-12
- Real-world test: Kimi K3.1 Pro costs 5.4x less than Opus 5.5 per task despite 2x list-price gap — Arindam_1729 · 2026-11-12