Paper: skill retrieval gains can hide worse performance on the very tasks retrieved
rohanpaul_ai · x · 2026-09-08
A paper uncovers a nasty evaluation failure in skill-enabled agents: positive retrieval gains can hide negative performance on the exact tasks where retrieval happened.
The flaw
- Most evaluations compare tasks where the agent chose to retrieve a skill against tasks where it didn't.
- These may be entirely different task types. If agents retrieve skills mainly on easier problems, retrieval looks helpful even when the skill did nothing — or hurt the answer.
The fix: RAE
- Whenever retrieval occurs, rerun that exact task without skill access and compare.
- This isolates the true causal contribution of skill retrieval on that task.
Related event: Paper Reveals Hidden Failure in Agent Skill Retrieval Benchmarks(2 posts)→
More from Research
- Mitra-v2: Synthetic-Data-Only 77M Tabular Model Matches 1.6B Rivals on TabArena — chaumian · 2026-09-08
- Sony AI's Hakken system turns 1.5M aging hypotheses into 2 confirmed gene discoveries — i_dg23 · 2026-09-08
- New worklog details building an async RL framework from scratch in JAX, from multi-actor systems to weight sync — yoshiyama_akira · 2026-09-08
- TASTE: A New Benchmark Testing If Models Can Predict AI Safety Researchers' Preferences — burny_tech · 2026-09-08
- EMNLP 2026 Paper: Training World Models for Behavior Consistency Cuts False Positives from 42.5% to 9.5% — 机器之心 · 2026-09-08
- Track4World: HKUST and Tencent ARC's feedforward model densely tracks every pixel in 3D — rsasaki0109 · 2026-09-08