30B tokens of data shows completion rates rise with task difficulty, not fall
Hamza_StrategizeLabs · reddit · 2026-10-22
Analyzing 600 substantial jobs and 30B tokens of activity, a platform found completion rates rise with difficulty: simple tasks land in the high 80s, while the most complex, highest-token jobs hit 98-100%. Addressing selection bias, the author notes abandonments cluster in low-stakes, vaguely specified, low-token requests—trivial tasks lack clear success conditions, so there is no success. Another surprise: across 1,900 deliverable sessions, 64.5% of output was kept with zero edits. Implication: testing AI on small tasks first evaluates it where your process is least defined; better to hand it a real, well-scoped job with human sign-off and judge against an existing standard.
More from AGI Musings
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- ISMS Fellow Avi Goldfarb: AI disruption raises returns to exploring research directions — avicgoldfarb · 2026-10-23
- signull on AI-found PHP exploits: economic liability is the forcing function legacy software needs — signulll · 2026-10-23
- TheStalwart: Talking computers are sparking a new philosophical boom, echoing ancient Greece — toptickcrypto · 2026-10-23
- AI cyberattacks could paradoxically accelerate AI development in an offense-defense arms race — kimmonismus · 2026-10-23
- Economist on AI detector Pangram: not a filter, but a useful verification signal — paulnovosad · 2026-10-23