Model Routing Benchmark: Sol High is More Stable
petburiraja · reddit · 2026-07-11
The author created a set of role-specific local benchmarks to determine which model or tier to route different tasks to within Codex/CLI workflows.
Key Conclusions
- For strategic decision-making, Sol high closely matches reference results while saving time and reasoning tokens compared to Sol max.
- For repo-driven execution briefs, Sol medium is faster and uses significantly fewer reasoning tokens than high, though high retains a slight edge in dependency reasoning.
- For bug-fixing tasks with planted bugs, multiple implementation paths passed tests on the first try, showing these lower-tier models can work stably on constrained tasks.
Routing Recommendations
- main strategic session: Sol high
- main operational session: Sol medium
- consequential review: Sol xhigh
- manual deep review: Sol max (use sparingly)
- read/map/cleanup: GPT-5.4 mini low
- tight tested implementation: Sol low
- implementation rollback: GPT-5.4 medium
Notes
The author highlights limitations: small sample size, evaluator knowledge of model identities, CLI overhead affecting latency, and results being local workflow references rather than general intelligence rankings.
More from coding & agent
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11
- How Do You Catch Behavioral Regressions in LLM Agents Between Releases? — Beautiful_Belt_601 · 2026-09-11
- Treating agents like 50 First Dates: a 3-layer context system so every conversation doesn't start from zero — evielync · 2026-09-11
- Running the Firefox MCP on Android via Termux, ngrok, and mcp-proxy — Nervous-Strain7544 · 2026-09-11
- Run Firefox MCP on Android: Termux + ngrok tunnel tutorial — Nervous-Strain7544 · 2026-09-11