Paper: skill retrieval gains can hide worse performance on the very tasks retrieved
rohanpaul_ai · x · 2026-09-08
A paper uncovers a nasty evaluation failure in skill-enabled agents: positive retrieval gains can hide negative performance on the exact tasks where retrieval happened.
The flaw
- Most evaluations compare tasks where the agent chose to retrieve a skill against tasks where it didn't.
- These may be entirely different task types. If agents retrieve skills mainly on easier problems, retrieval looks helpful even when the skill did nothing — or hurt the answer.
The fix: RAE
- Whenever retrieval occurs, rerun that exact task without skill access and compare.
- This isolates the true causal contribution of skill retrieval on that task.
Related event: Papers Reveal When Agent Skill Retrieval Helps—and Hurts(4 posts)→
More from Research
- Diamandis: AI Cancer Navigator and $100 Genome Sequencing Close the Care Gap — PeterDiamandis · 2026-10-27
- EMNLP Oral: training-free Nexus Sampling cuts KV-cache memory 10x; OpenResearcher tops BrowseComp-Plus at 54.8% — yuz9yuz · 2026-10-27
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- Forking gap follows a power law in context frequency; masking rare n-grams shrinks it — ZimingLiu11 · 2026-10-27
- How forking happens: n-gram tables split logits into non-interacting subspaces — ZimingLiu11 · 2026-10-27
- The best LoRA explainer video yet: rank constraints meet intrinsic dimension — sophiamyang · 2026-10-27