antirez: LLM benchmarking is broken—we need jointly lab-funded trusted third parties
antirez · x · 2026-11-19
antirez argues that LLM benchmarking is broken because there are no trusted third parties running evaluations. His proposed fix: benchmarks run by organizations jointly funded by AI labs—not VCs—that primarily run benchmarks where models compete against each other on tasks.
More from Models
- Heavy testing finds Astra 6.2 High slightly worse than 6.0 and 2x slower — IridiumEagle · 2026-11-19
- Heavy Astra 6.2 testing still can't exhaust the $100 plan's usage quota — prd_008 · 2026-11-19
- Gemini 4 Carbon Deciphers an Unread 1466 Sforza Renaissance Cipher Unaided — fofrAI · 2026-11-19
- Gemini's mysterious "Hydrogen" selector appears to route to Gemini 3.1 Pro — Neestor00 · 2026-11-19
- Fable 5.2 vs GPT-6.2 Astra: 108M vs 24M tokens to build the same PS1-style game — heypearlai · 2026-11-19
- Same game prompt: Astra ships for $17 vs Fable's $65 — 3x cheaper, fewer bugs — heypearlai · 2026-11-19