Why 'Full Pass Rate' is a flawed metric for LLM evaluation
xeophon · x · 2026-08-21
The author criticizes 'Full Pass Rate' as a metric, arguing it depresses scores, offers little insight into progress, and requires flawless graders—which they have yet to see.
More from Models
- SenseTime Open Sources SenseNova U1.5 Lite: 8B Model for Native 4K Generation and Precise Image Editing — aftahi_ai · 2026-08-21
- User reports exhausting GPT-5.3-Codex-Spark quota in 20 minutes on Pro plan — nijfranck · 2026-08-21
- Gemini 3.7 Flash hits 84.6% on ARC-AGI-2 at just $0.25 per task — fofrAI · 2026-08-21
- Musk Confirms Work to Improve Grok's Writing Skills — mark_k · 2026-08-21
- ARC Prize Adds Model Comparison, Gemini 3.7 Flash Scores High — mhmazur · 2026-08-21
- Anthropic's Fable Breaks RareBench Record After Relaxing Filters — danielmckinn0n · 2026-08-21