Model Routing Benchmark: Sol High is More Stable
petburiraja · reddit · 2026-07-11
The author created a set of role-specific local benchmarks to determine which model or tier to route different tasks to within Codex/CLI workflows.
Key Conclusions
- For strategic decision-making, Sol high closely matches reference results while saving time and reasoning tokens compared to Sol max.
- For repo-driven execution briefs, Sol medium is faster and uses significantly fewer reasoning tokens than high, though high retains a slight edge in dependency reasoning.
- For bug-fixing tasks with planted bugs, multiple implementation paths passed tests on the first try, showing these lower-tier models can work stably on constrained tasks.
Routing Recommendations
- main strategic session: Sol high
- main operational session: Sol medium
- consequential review: Sol xhigh
- manual deep review: Sol max (use sparingly)
- read/map/cleanup: GPT-5.4 mini low
- tight tested implementation: Sol low
- implementation rollback: GPT-5.4 medium
Notes
The author highlights limitations: small sample size, evaluator knowledge of model identities, CLI overhead affecting latency, and results being local workflow references rather than general intelligence rankings.
More from coding & agent
- charlie Marsh: rewrites beat external deps in the AI era — mohamedmansour · 2026-10-26
- OpenAI and Twitch launch Codex hackathon with $25,000 in prizes — OpenAIDevs · 2026-10-26
- Filtering secrets on the write path: why agent memory deserves more suspicion at ingest than retrieval — Classic-Being-7960 · 2026-10-26
- INT21's AI agent swarm rewrote Drama 3's inference runtime in Rust within one hour, beating SGLang-Omni — bingxu_ · 2026-10-26
- After OpenCode key leak: 5 practical rules for keeping keys away from agents — jacobholm_dev · 2026-10-26
- wandb 0.31.0 ships with 2x faster startup, adds JSON output for coding agents — wandb · 2026-10-26