Models don't practice what they preach: Opus 5.5's code calls its top-recommended API only 4% of the time
santiviquez · x · 2026-11-10
santiviquez highlights a counterintuitive quirk: ask a model to recommend an API for a goal versus ask it to use an API for the same goal, and you get completely different answers.
The accompanying chart compares whether each model's own code actually calls the API it recommends most: GPT-5.6 Sol 63% (30/48), GPT-6 Astra 52% (25/48), Fable 5.1 50% (24/48), and Opus 5.5 just 4% (2/48). Most models' coding behavior diverges sharply from their stated recommendations.
More from Models
- Together Compiler hits 2.7x throughput gains on Kimi K3 at same per-user speed — togethercompute · 2026-11-10
- Surge AI Benchmark: Higher Thinking Effort Beat Every Coding Agent Harness Tested — rseroter · 2026-11-10
- Claude Fable 5.2 tops WebDev Arena with 1832 pts, prices cut to $8/$40 — arena · 2026-11-10
- Lightweight Muse Ember one-shots an interactive 3D tea garden, Meta reshares test — AIatMeta · 2026-11-10
- Uncensored gemma-4-31B heretic GGUF hits Hugging Face trending — mradermacher · 2026-11-10
- Fable 5.2 still loses to GPT-6 Astra at Blood on the Clocktower, but beats Opus 5.5 — teortaxesTex · 2026-11-10