Reviewer recomputes all 7,000 benchmark answers — then finds the $38,320 sausage
ziv_ravid · x · 2026-11-10
A diligent ICLR reviewer recomputed all 7,000 reference answers of a benchmark paper on reading numbers across cultures (GLOBENUM) — every one matched. Then they actually read the questions: items like "Each sausage cost 38,320 Dollars" (38.32 under the decimal reading) and "her 433lb dog." Verdict: clear rejection. Lesson: consistency checks won't catch benchmark items that are semantically absurd.
Related event: Reviewer Verifies 7,000 Benchmark Answers, Finds Absurd Questions(2 posts)→
More from Fun
- Fable 5.2 vs GPT-6 Astra at xhigh reasoning: stunning isometric pixel-art generation — Angaisb_ · 2026-11-10
- Fable 5.2 builds a real-time 'Mirror Earth' game mirroring real-world weather — mattshumer_ · 2026-11-10
- Ethan Mollick had Fable 5.2 build a full reverse Scooby Doo game with its own trailer — emollick · 2026-11-10
- Joke video shows Jensen Huang 'escaping his handlers' en route to Megaspeed QBR — julsimon · 2026-11-10
- Vibe-codevember day 8: WebGPU tesseract built with Gemini 4 and Antigravity — AI_Andrew · 2026-11-10
- Fable 5.2 builds a playable voxel engine from a single prompt — mattshumer_ · 2026-11-10