Long-context models still copy irrelevant text, and a new reward lifts accuracy by up to 4.6 points
rohanpaul_ai · x · 2026-07-23
Long-context models still waste tokens by copying irrelevant text, and a new reward fixes it
A paper on long-context reasoning finds that even strong models often fall into repetitive copying: they copy large chunks of the prompt into their internal reasoning instead of focusing on the key evidence.
- The problem gets worse as input length grows.
- Copying the right lines helps; copying surrounding filler hurts accuracy.
- The authors propose GEAR (Grounding Evidence-Aware Reward), a reward-shaping method that boosts attention to key evidence and penalizes overlap with irrelevant context.
- On natural-language data, they also build an automated pipeline that creates evidence-annotated training data from arbitrary documents.
- Their experiments report up to 4.6 points of accuracy improvement over standard RL rewards, along with less repetitive copying and shorter thinking traces.
The paper’s takeaway is that long-context reasoning still depends on precise grounding, not just bigger context windows.
Related event: Study Reveals Long-Context Models Waste Tokens Copying Text(2 posts)→
More from Research
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- FLASH: GPU-native deformable sim learns cloth folding in minutes on one RTX 5090 — ChongZzZhang · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- Behind the EMNLP keynote: LMs as simulated learners to generate child language acquisition hypotheses — najoungkim · 2026-10-26
- The paper behind the EMNLP 2026 keynote on bigram 'simulated learners' — najoungkim · 2026-10-26