Long-context models still copy irrelevant text, and a new reward lifts accuracy by up to 4.6 points

rohanpaul_ai · x · 2026-07-23

Long-context models still waste tokens by copying irrelevant text, and a new reward fixes it

A paper on long-context reasoning finds that even strong models often fall into repetitive copying: they copy large chunks of the prompt into their internal reasoning instead of focusing on the key evidence.

The problem gets worse as input length grows.
Copying the right lines helps; copying surrounding filler hurts accuracy.
The authors propose GEAR (Grounding Evidence-Aware Reward), a reward-shaping method that boosts attention to key evidence and penalizes overlap with irrelevant context.
On natural-language data, they also build an automated pipeline that creates evidence-annotated training data from arbitrary documents.
Their experiments report up to 4.6 points of accuracy improvement over standard RL rewards, along with less repetitive copying and shorter thinking traces.

The paper’s takeaway is that long-context reasoning still depends on precise grounding, not just bigger context windows.

Original post →

Long-context models still copy irrelevant text, and a new reward lifts accuracy by up to 4.6 points

More from Research