Benchmark: model scoring 2/3 on standard case drops to 0/3 when budget tightens
ML_Felix · reddit · 2026-10-28
The Concourse Bench v1 benchmark ran eight AI models through the same multi-step task — a simulated media buy with negotiation against three fixed-rule sellers. Aggregate scores hide the real story: Sol and Fable went 3/3 in every scenario, while Terra scored 2/3 on the standard case but 0/3 with a tighter budget, 0/3 against firmer sellers, and 1/3 with more inventory. The author (disclosing they work at the publishing company) asks how practitioners stress-test models beyond happy paths.
More from Models
- Zero-training PCA head lifts 577M EmbeddingGemma-2 to #1 on MTEB audio, beating 9B models — JeanMichelRanu · 2026-10-28
- Arabic course creator weighs Opus 5.5 against discounted Gemini 4 Argon at $6 — Various_Specific_623 · 2026-10-28
- Google bans user from Gemini, AI Studio and Search after repeated safety-filter flags — immellocker · 2026-10-28
- Richard Socher's Recursive open-sources Flint: 40B MoE, 2.5B active, Apache 2, runs coding locally on Mac — altryne · 2026-10-28
- vLLM Semantic Router and AMD release Vela 3.0: 380M model beats GPT-5-mini at zero-shot — vllm_project · 2026-10-28
- User stuck with €104 of accidentally bought, unused ChatGPT credits and bot-only support — NickTsl · 2026-10-28