DIY 12-task real-work benchmark: Opus 5.5 most reliable, Muse tops average but hangs
macaronianddeeez · reddit · 2026-10-14
Tired of public leaderboards, the author built a personal benchmark from real work: 12 task types (bug fixes, features, doc research, live web research, client inbox triage, payment reconciliation, specs) with hidden checks models never see — hidden tests, planted instructions, fake email/calendar logging. Serious mistakes cap a task at 20; flaky tasks repeated 3-4x.
Scores (avg / live research): Muse Wonder 98/84 (single run; Contributor mode means Meta trains on your data), Sol 6.1 High 96/91, Astra Medium 96/96 (perfect on bug fixes), Opus 5.5 90/98, Fable 5.1 88/87.
Findings:
- Opus and Fable confidently assigned an unidentified $240 payment to a client instead of flagging it — capped at 20; without it they'd average 96/94
- Sol and Astra silently changed booking-form behavior on client-facing work
- Opus wrote the best research answer but read local files despite a web-only instruction (read-only, flagged)
- Muse is capable but least stable: hangs forever on tests that never exit, one task took 41 minutes
Final workflow: Sol 6.1/Astra Medium (sandboxed) for bugs and long jobs, Astra for numbers, Opus 5.5 for research/writing/project continuity, Muse as supervised overflow.
More from coding & agent
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23
- DeepLearning.AI weekly: GPT-6.1 hits 97.9% byte parity on COBOL migration — DeepLearningAI · 2026-10-23