256 real mobile jobs: Claude Haiku 5.5 ties GPT-6 Luna despite 23.5-point benchmark gap
LovesWorkin · reddit · 2026-10-14
Buoy's lead dev ran a third-party eval via Ask Buoy: while Anthropic's chart shows Haiku 5.5 beating GPT-6 Luna by 23.5 points on OSWorld 2.1 (desktop computer use), the two models tied on 256 real mobile app jobs (iOS Simulator, same food-ordering test app, default settings, counted only if the app showed the result).
- Correct jobs: Haiku 93.8% vs Luna 94.1%
- First answer: Haiku 1.6s vs Luna 2.9s — Haiku faster
- Whole job (median): Haiku 20.9s vs Luna 27.9s
- Cost per job: Luna 0.44¢, 29% cheaper than Haiku's 0.62¢
- Haiku led on long bug stories (94 vs 88); Luna led on web calls, saved data, location, undo
Caveats: one run per job, jobs written in-house, models tested at different times, cost from list-price tokens. Takeaway: benchmark charts can diverge sharply from real workloads.
More from coding & agent
- The agentic design workstation: designing with real code, from canvas to PR — floguo · 2026-10-14
- RAG Isn't Dead After All, Developer Concludes After Real-World Use — jakedahn · 2026-10-14
- Agent team study: 84% of messages still go to the lead, workers rarely talk to each other — nouhadziri · 2026-10-14
- Sol lead + 16 cheap Luna planners nears all-Sol team results at 70% less cost — nouhadziri · 2026-10-14
- Delegate Only Planning: Agent Team Matches Full Delegation at Half the Cost — nouhadziri · 2026-10-14
- Agent Teams Beat Solo by Up to ~10 Points but Cost 4–7x More — nouhadziri · 2026-10-14