177-case eval: cheapest LLM matched models costing 5–11x more for task parsing
danielbgolan · reddit · 2026-10-24
A developer building an iOS to-do app created a 177-case eval set for task parsing (multilingual text, screenshots, prompt-injection emails, edits, multi-task splitting), ran 4 models via OpenRouter with 3 runs per case plus blind dual-judge scoring.
Results (quality = judge avg; cost/1k; median time)
- GPT-6 Luna: 7.73–7.88, 174–175/177, $0.23, 3.0–3.6s
- DeepSeek V4.1 Flash: 7.88, 172/177, $1.15
- Claude Haiku 5.5: 7.80/$1.09; high reasoning 7.93/$1.45; xhigh 7.93, 177/177, $1.91
- GPT-5.4 mini: 7.63, 172/177, $2.53
Takeaways
- Run-to-run noise rivals model gaps (Luna ranged 7.73–7.88)
- The cheapest model was 11x cheaper than GPT-5.4 mini with slightly better quality
- Higher reasoning gained 7.80→7.93 for 2x latency — not worth it when users wait
- Prompt examples beat length: none 7.69, half 7.93
- Count provider errors: Haiku hit 69 in one run, Luna zero
More from Models
- Kimi K3.1's first impressions spark buzz: 'smells like Claude' — MatthewBerman · 2026-10-25
- OrcaRouter Releases Abliterated DeepSeek V4.1 Flash With 1M Context for Verified Security Researchers — brhydon · 2026-10-25
- AI is the security debt collector: exploits now found in hours — wunderwuzzi23 · 2026-10-25
- Frontier intelligence needs Flash-level speed: tps beats benchmark-tuned models in daily work — _arohan_ · 2026-10-25
- Aryan asks: Are Kimi or GLM as good as Sol or Astra in daily work? — _arohan_ · 2026-10-25
- Long-time ChatGPT user: Claude feels like a mentor, Codex like a frustrating machine — aziz4ai · 2026-10-25