RL evals have a cost problem: expensive tests cap your sample sizes
jessi_cata · x · 2026-11-21
jessicata points out an often-overlooked constraint in RL training: the cost of the tests themselves. Some capabilities are verifiable — e.g. checking whether an AI completed a pre-training run at a given cross-entropy — but such verifications can be extremely expensive, directly limiting feasible RL sample sizes.
Related event: Researcher Warns Evaluation Cost Limits RL Sample Sizes(2 posts)→
More from Research
- Vik Sikka on the reported Claude-assisted proof of the Riemann Hypothesis: 'an incredible day' — vsikka · 2026-11-21
- MIT's Sutherland Verifies Riemann Hypothesis Lean Proof: All 175,187 Modules Compile — altryne · 2026-11-21
- Live Proof Monitor Launched for Independent Lean Verification of Riemann Hypothesis Proof — altryne · 2026-11-21
- DL researcher: we lack a theory of deep learning because you can't hill climb on it — jasondeanlee · 2026-11-21
- EFE-98 sim benchmark hits 0.98 correlation with real-robot rankings at 1/10 the cost — ManlingLi_ · 2026-11-21
- Terence Tao blogs on the Riemann hypothesis, weighs AI-generated proofs and Masullo manuscript — burny_tech · 2026-11-21