The Marshmallow AI Benchmark: Testing vision models with counting
qu1etus · reddit · 2026-08-21
The author created the "Marshmallow Benchmark": a photo of marshmallows on a baking sheet, asking AI to count them accurately without guessing. Despite strict prompts, major models (Gemini, Claude, GPT) gave varied results ranging from 472 to 539. It's a fun test highlighting vision model capabilities regarding detail counting and hallucinations.
More from Multimodal
- fal launches Worlds for real-time creation and interaction with AI-generated worlds — Scobleizer · 2026-10-16
- Gummy worms as a physics test: Gemini 4 Argon vs. Claude Opus 5.5 elasticity sims — DynamicWebPaige · 2026-10-16
- 260M-parameter looped diffusion transformer beats 6x larger baselines with 5x less compute — burkov · 2026-10-16
- Seedance 2.5 demo: stretching one second of an explosion into a 30-second shot — LudovicCreator · 2026-10-16
- Octopus Short Film Made with Grok Imagine, Upscaled via Topaz — creatoroff · 2026-10-16
- Open-sourced 61-second computing history film generated entirely by Muse Spark — sophiamyang · 2026-10-16