A power-law task difficulty model explains why equal capability gains produce uneven eval improvements
teortaxesTex · x · 2026-10-31
Eval-focused commentator teortaxesTex sketches a compact model of task complexity: difficulty is continuously distributed with a power-law tail, and a capability improvement multiplies every task's success odds by the same constant factor (2.73x).
The attached derivation shows a reproducible construction:
- Success probability follows a noisy-threshold sigmoid; raising capability c scales every task's odds equally.
- Near-saturated tasks see large pass@1 gains but little pass@3 headroom; extremely hard tasks improve proportionally yet remain very unlikely to pass within three attempts.
- Fitting the power-law difficulty distribution reproduces uneven gains (pass@1 88.10%→92.30%, pass@3 94.82%→95.36%) with no task-specific memorization mechanism and no permanent error floor.
The author remains skeptical about RL's effect on near-impossible tasks but offers this as a neat mental model for evals.
More from Research
- After NVIDIA's BindClaw, every AI4Bio paper now needs an agentic pipeline demo to prove versatility — rishabh16_ · 2026-10-31
- Neural Thickets Revisited: Frozen-Model Logit Perturbations Match Full-Weight Perturbation Ensembles — kastnerkyle · 2026-10-31
- CS312 course post explains sharpness and the edge of stability in LLM pretraining — tatsu_hashimoto · 2026-10-31
- AI decodes wolf howls: 500K hours of Yellowstone audio analyzed by models — aza · 2026-10-31
- NBER paper: training your AI replacement can actually raise workers' pay — joshgans · 2026-10-31
- Trata's Hedge-Bench: no model tops 50% on real-world financial forecasting — ycombinator · 2026-10-31