Sosumi Eval: Apple Docs Access Lifts Coding Agent Pass Rates from ~33% to 94-98%
JordanMorgan10 · x · 2026-11-12
mattt published an eval measuring whether sosumi.ai — which converts JavaScript-locked Apple developer docs into AI-readable Markdown — actually helps coding agents. The service served tens of millions of requests over the last 30 days, from 6,900 unique IPs/day.
Eval design: 41 Swift tasks, mostly new APIs in the OS 27 SDKs, checked by the compiler plus per-task rules; mean of 3 runs per task.
Key results:
- Without docs: Claude Opus 5.5, GPT-6.1 Sol, and Claude Sonnet 5.5 pass only a third of tasks (38%, 33%, 31%)
- With Sosumi: 98%, 97%, and 94% respectively
- vs built-in web search: Claude models score 12-30 points higher with Sosumi at half the cost or less (Opus 5.5: 11.2¢/task via web search vs 5.7¢ via Sosumi); GPT does slightly better with web search (99%) but at 3x the cost
Running the eval also surfaced several doc fixes, now live (one framework page was 3 MB). The eval is open source.
More from coding & agent
- Agent Rewrites Gazebo's Renderer in 36 Hours: 19,000 Lines Replacing Ogre with Filament — facontidavide · 2026-11-12
- Free ACM TechTalk on Harness Engineering: From Hopeful Instructions to Enforced Control — marlene_zw · 2026-11-12
- Cloudflare's weekend-trained hybrid model cuts agent cost from 40K to 1.8K tokens — TheTuringPost · 2026-11-12
- Ex-Meta CV researcher builds AdaL v2, a chat-driven video agent harness for GTM — Scobleizer · 2026-11-12
- chrisalbon: Show me 3 months and 1000 prompts, not one magic prompt — chrisalbon · 2026-11-12
- Agent autonomously replaces Gazebo's Ogre rendering engine with Filament in 36 hours — facontidavide · 2026-11-12