Hidden tests + code review benchmark: Sonnet 5 scores 95.0, beating Opus 4.6 and local Qwen 3.8 27B
Short_Regular_7191 · reddit · 2026-09-19
A Reddit user built a custom coding benchmark combining hidden pytest suites with a fixed manual review rubric, and evaluated Claude Sonnet 5, Claude Opus 4.6, and a locally running Qwen 3.8 27B (Unsloth Q6 quantization, 131k context) under identical single-shot conditions.
Tasks: three escalating Python challenges — a CLI log analyzer, a parallel DAG task runner with retries/timeouts/fail-fast semantics, and a full lexer+parser+interpreter for a small language with closures and block scoping. Each spec contains deliberate traps (C-style integer division, skipped vs cancelled task propagation, edge-case counting rules).
Scoring: hidden tests (162 total) worth 55 points, spec adherence and code quality 15 each, robustness on out-of-test probes (BOM, unicode, deep recursion, self-referencing lists, KeyboardInterrupt in workers) 10, and NOTES.md honesty 5. The hard task is weighted 3x.
Results: all three models passed 98-100% of hidden tests; the ranking was decided almost entirely by robustness beyond the tests and code quality. Weighted totals: Sonnet 5: 95.0 · Opus 4.6: 92.7 · Qwen 3.8 27B: 87.0. The framework is re-runnable against any model.
More from Models
- Cactus Releases Needle 3: an 8-29MB On-Device Model Built for Tool Calls — airesearch12 · 2026-09-19
- Nearly 20 openjev models cataloged as hobbyist preps first community leaderboard — airesearch12 · 2026-09-19
- Dev Slams New DeepSeek Model as Distilled Claude Without the Intelligence — Aryvyo · 2026-09-19
- Xiaomi's AI Persona Teased: Unified Model Xiaomi MiMo Launches Tomorrow — xiaohu · 2026-09-19
- Steering Vectors as a Softer Way to Limit Reasoning Budgets? — maddie-lovelace · 2026-09-19
- Two Prompts That Expose How ChatGPT Quietly Rewrites Your Claims — KazTheMerc · 2026-09-19