"24 circles to draw 24" minimal test: Gemini fails while Opus and ChatGPT pass

On Oct 11, a minimal test circulated on X: ask a model to draw the number 24 using exactly 24 black circles on a white background, no more, no less. Testers ran the same prompt across models with clearly divergent results, with Gemini judged by several as a "second-tier" model.

Confirmed

Safety researcher David Manheim confirmed Gemini failed even in high thinking mode: it gave up before claiming to reach 24 and miscounted the circles, leading him to call Gemini clearly second-tier relative to frontier models.
Manheim reported Opus 4.6 passed without issue, as did Opus 5.5 (Medium) and ChatGPT; he added that even Haiku 4.5 counted correctly, arguing a "decent model" would only fail if the task were made far harder.
Tester @upperx ran the same minimal prompt on Gemini (high): the generated image had the wrong number of circles, and the model still answered incorrectly when asked how many it drew, pointing to weak self-verification of its own output.
Tester DellAnnaLuca initiated/joined the discussion concluding Gemini is second-tier while Opus and ChatGPT pass easily; Manheim added the Haiku 4.5 comparison.

Why it matters

This is a low-cost, reproducible community benchmark: a single minimal prompt separates models on counting and self-verification, a quick way to probe model weaknesses.
The results show a perceptible gap between Gemini and Claude/ChatGPT-series models on multimodal counting, and an inability of the image model to verify its own just-generated output.
Manheim also cautioned on interpretation: failures on low-tier models say more about the poster's unfamiliarity with high-tier models than about LLM capabilities in general.

2026-10-11 ~ 2026-10-11 · 6 related posts

Primary sources

Even Haiku 4.5 Passes This Reasoning Test, as Opus 4.6 Counters Model-Gap Claims — davidmanheim · 2026-10-11 02:37 (UTC+08:00)
'Draw 24 with exactly 24 circles' test: Gemini fails, Opus 5.5 and ChatGPT pass — davidmanheim · 2026-10-11 02:29 (UTC+08:00)
Asking an image model to draw 24 with exactly 24 circles still fails — upperx_ · 2026-10-11 02:01 (UTC+08:00)

[source] Asking an image model to draw 24 with exactly 24 circles still fails — upperx_ · 2026-10-11
[source] 'Draw 24 with exactly 24 circles' test: Gemini fails, Opus 5.5 and ChatGPT pass — davidmanheim · 2026-10-11
Same reasoning puzzle: Gemini flubs it while Opus 5.5 and ChatGPT pass easily — DellAnnaLuca · 2026-10-11
Gemini miscounts and gives up early in community reasoning test — davidmanheim · 2026-10-11
Researcher's Counting Test: Gemini Gives Up Early, Opus 4.6 Aces It — davidmanheim · 2026-10-11
[source] Even Haiku 4.5 Passes This Reasoning Test, as Opus 4.6 Counters Model-Gap Claims — davidmanheim · 2026-10-11