NVIDIA paper: model accuracy drops 62.8% as context grows from 4K to 128K on long agent tasks
dair_ai · x · 2026-10-02
A new NVIDIA paper on long-running agents introduces Long-Transduction, a benchmark setup where models must keep reading, updating, and emitting state-dependent outputs over thousands of tokens, varying three factors independently to isolate what breaks long tasks.
Across seven open-weight models, accuracy drops 62.8% when context grows from 4K to 128K, 36.5% from input format changes alone, and 39.9% when per-step operations get harder. The takeaway: a model accepting 128K tokens still makes more mistakes the longer it works—explaining why agents lose their place mid-way through long tables or ledgers.
Related event: NVIDIA Paper: Model Accuracy Drops 62.8% on Long Tasks as Context Grows(2 posts)→
More from coding & agent
- AI in finance shows a power law: top 1% users generate 35x more files with agents — peterjliu · 2026-11-19
- Linux LTS patch flood: 3318 patches for 6.6 alone, 'no way humans are reviewing this' — pkqzy888 · 2026-11-19
- Caliper Harness: open-source coding harness disciplines small local models with plans, checks, reviews — Saraozte01 · 2026-11-19
- AI systems lack executive function, not scale: four manager-layer strategies to fix it — Designer-Material633 · 2026-11-19
- Google employee replaces doomscrolling with 3 fully automated personal AI newsletters — DynamicWebPaige · 2026-11-19
- Designer spins up an agent to build state-airline brand gallery mid-flight delay — beh_zod · 2026-11-19