Simplifying the Prompt Crashed GPT-5.4 From 93.5% to 40.9% in Eval Write-Up
dl_weekly · x · 2026-11-03
DL Weekly's cautionary evaluation piece on novelty judging shows how fragile eval pipelines can be: simplifying the prompt alone dropped GPT-5.4 from 93.5% to 40.9% on the same task. Meanwhile, a plain Claude Opus 4.6 baseline beat purpose-built systems at $2.59 per run versus $82.05 — roughly 31x cheaper. The takeaways: complex multi-model systems can lose to a well-tuned baseline, and eval results are highly sensitive to prompt wording, so benchmarks should always document their prompt versions.
More from Models
- Google AI Chat Long Conversations Break, Only Returning Error Message — batimfan4ever · 2026-11-04
- Open-source FM-Pochi-32B claims IMO 2026 bronze, first fully open LLM to medal — Ryan__Rhys · 2026-11-04
- Testing LongCat-2.5 on Real Long-Horizon Work via Claude Code — FellMentKE · 2026-11-04
- LongCat 2.5 Tested in Claude Code: One Prompt Builds a Cinema Landing Page — FellMentKE · 2026-11-04
- 'Astra Can Read and Write Weights to Improve Qwen' Claim Debunked as Misleading — AlyoshaV · 2026-11-04
- JFPuget's team takes 3rd on both Kaggle ARC-AGI competitions ending the same day — JFPuget · 2026-11-04