NVIDIA paper: model accuracy drops 62.8% on 128K-token tasks vs 4K
rohanpaul_ai · x · 2026-10-02
A new NVIDIA paper tested 7 open models on simple repetitive tasks (adding numbers, sorting lists) and found reliability degrades sharply with task length even within the context window.
Key findings:
- Average accuracy was 62.8% lower on 128K-token jobs than 4K-token ones
- Even the best model got every item right in only 17.1% of the longest jobs
- Models seem to understand the task but lose their place, especially when items lack IDs
Practical advice for agent builders: number every item, split big jobs into small batches, and verify every line of output.
Related event: NVIDIA Paper: Model Accuracy Drops 62.8% on Long Tasks as Context Grows(2 posts)→
More from coding & agent
- Claude Code overtakes Cursor and Copilot in job-post mentions ahead of Anthropic IPO — antgoldbloom · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- AWS adds private CA support to Bedrock AgentCore Identity for connecting agents to self-hosted IdPs — AWS ML Blog · 2026-11-19
- Orchestrator's Cloud Agents graduate from beta to general availability — julianweisser · 2026-11-19
- Zero users, 33 pages shipped: dev crowdsources audit for AI-agent trust schema — TheHamzaShahbaz · 2026-11-19
- She vibecoded her own photobooth with Claude Code — and shared the prompt — toolstelegraph · 2026-11-19