Ox Alpha scores ~63% on full DeepSWE; earlier 80% was subset artifact
Full DeepSWE results for the mysterious model Ox Alpha have landed: @davis7 measured 63%, while an independent tester got 58.4% (66 of 113 tasks), essentially matching Claude Opus 4.8's 59%. The circulating 80% pass rate was an artifact of subset testing. Ox Alpha's real coding ability is roughly on par with GPT-5.6 Sol mid, DeepSeek V4 Pro and Grok 4.6 — first-tier, but not a runaway leader.
Confirmed
- @davis7's full DeepSWE run scored 63%, not the 80% from the earlier subset test; he found its "tone" much better than Claude or GPT, with decent design.
- An independent run on the full 113-task benchmark yielded 58.4%, nearly identical to Claude Opus 4.8's 59%; commenters see it as a good sub-agent.
- @zainhas's evaluation showed average output of 47k tokens, making it Pareto-optimal among open models and close to closed-source Grok 4.6.
- @kimmonismus speculated the model could be GLM-5.3 Flash; unverified.
Why it matters
- The inflated 80% figure drew huge attention; full runs correct the overestimation and warn against trusting small-sample benchmarks.
- Even with agent coding back in a realistic range, Ox Alpha's token efficiency and tone are seen as standout strengths, making it a notable new open-model option.
2026-08-22 ~ 2026-08-23 · 6 related posts
- Episode 1: Mystery model Ox Alpha stuns community; fingerprints point to Zhipu GLM-5.3(2026-08-21, 57 posts)
- Episode 2: OpenRouter's Mystery 'Ox Alpha' Model Identified as GLM(2026-08-21, 3 posts)
- Episode 3: Cola Launches Ox Model Claiming Terra-Level Capability at Flash Speed(2026-08-21, 2 posts)
- Episode 4: Mysterious Model Ox Alpha Gives Away 100T Tokens Daily, Sparking Power Questions(2026-08-21, 2 posts)
- Episode 5: Ox-Alpha Underperforms in Tests, Called Marketing Stunt(2026-08-22, 3 posts)
- Episode 6: Ox Alpha scores ~63% on full DeepSWE; earlier 80% was subset artifact(2026-08-22, 6 posts)
- Episode 7: Forensics Point to Zhipu's Unreleased GLM Behind Mystery Model Ox Alpha(2026-08-22, 31 posts)
- Episode 8: Ox Alpha Generates 64K-Token 3D Scene in One Prompt(2026-08-22, 2 posts)
- Episode 9: Mystery model Ox Alpha draws mixed reviews from early testers(2026-08-22, 6 posts)
- Episode 10: Mystery Model Ox Alpha Launches Free, Identity Unknown(2026-08-23, 3 posts)
- Episode 11: Mysterious Model Ox Alpha Debuts on OpenRouter, Sparking Speculation(2026-08-24, 5 posts)
- Episode 12: Leak: Ox Alpha Is GLM-5.3 Flash, 320B MoE at One-Tenth the Price(2026-08-25, 4 posts)
- Episode 13: Mystery Model Ox Alpha Revealed as Zhipu's GLM-5.3 Flash, Weights Going Open-Source Tonight(2026-08-25, 17 posts)
- Episode 14: Stealth Model Ox Alpha Proves Mediocre at Coding, Trails Grok 4.6(2026-08-25, 3 posts)
- Episode 15: Mystery Model Ox Alpha Revealed as Zhipu's GLM-5.3-Flash, Served on Domestic Chinese Chips(2026-08-26, 19 posts)
- Episode 16: Mystery Model Ox Alpha Burns 42T Tokens in Six Days(2026-08-26, 3 posts)
Primary sources
- [source] Open source model ox-alpha hits 63% on DeepSWE, rivals Grok efficiency — zainhas · 2026-08-22
- [source] Ox-alpha DeepSWE benchmark reveals 58.4% pass rate, debunking 80% rumors — apples_jimmy · 2026-08-22
- ox-alpha scores 58.4% on DeepSWE benchmark, matching Claude Opus 4.8 — AccBalanced · 2026-08-23
- [source] Ox Alpha mystery model scores ~63% on full DeepSWE, on par with GPT-5.6 Sol mid — kimmonismus · 2026-08-23
- Mystery model Ox Alpha scores 63% on full DeepSWE run, not the earlier 80% — daniel_mac8 · 2026-08-23
- Ox Alpha scores 63% on DeepSWE benchmark with mixed user feedback — AccBalanced · 2026-08-23