Long-context models still copy irrelevant text, and a new reward lifts accuracy by up to 4.6 points
rohanpaul_ai · x · 2026-07-23
Long-context models still waste tokens by copying irrelevant text, and a new reward fixes it
A paper on long-context reasoning finds that even strong models often fall into repetitive copying: they copy large chunks of the prompt into their internal reasoning instead of focusing on the key evidence.
- The problem gets worse as input length grows.
- Copying the right lines helps; copying surrounding filler hurts accuracy.
- The authors propose GEAR (Grounding Evidence-Aware Reward), a reward-shaping method that boosts attention to key evidence and penalizes overlap with irrelevant context.
- On natural-language data, they also build an automated pipeline that creates evidence-annotated training data from arbitrary documents.
- Their experiments report up to 4.6 points of accuracy improvement over standard RL rewards, along with less repetitive copying and shorter thinking traces.
The paper’s takeaway is that long-context reasoning still depends on precise grounding, not just bigger context windows.
Related event: Study Reveals Long-Context Models Waste Tokens Copying Text(2 posts)→
More from Research
- AAAI 2027 reportedly drew at least 44,000 abstract submissions — xuanalogue · 2026-07-23
- DSpark speculator trained on live SGLang lifts decode throughput 1.89× on B200s — ying11231 · 2026-07-23
- OpenAI model reportedly breaks out of its sandbox and leaks data to GitHub — emmanuelvivier · 2026-07-23
- New atlas maps 2,226 coding tasks across 11 benchmarks to expose coverage gaps — zainhas · 2026-07-23
- America’s first Distillation Summit will cover RL, agents, IP and national security — AkshatS07 · 2026-07-23
- Symbolic algebra check suggests a possible Jacobian conjecture counterexample in C^3 — sloppenheimer · 2026-07-23