METR time horizons use low-context humans; repo maintainers are 5-18x faster
BasilHalperin · x · 2026-11-09
BasilHalperin explores why OpenAI's internal time-horizon levels look low: METR measures against "low context" humans unfamiliar with the work. The METR paper's appendix estimates high-context repo maintainers are roughly 10x faster at completing the same issues (the table shows 5-18.6x across tasks).
Measuring against high-context humans would lower AI time horizons 10x, possibly explaining the low levels in OpenAI data — though not fully closing the gap. An alternative explanation is simply cross-domain variation: different task domains have different intercepts.
More from Research
- LoRA-tuned Qwen3 skips first output line whenever the caption is under ~4 tokens — prathamdmehta · 2026-11-09
- Ramez: OpenAI's Productivity Gain Per ECI Point Appears to Decline Over Time — ramez · 2026-11-09
- Betting on transformers: incremental improvements will beat radically new LLM architectures — max_paperclips · 2026-11-09
- FragBench benchmark paper shared, details in the paper itself — arthurcolle · 2026-11-09
- Economist Paul Novosad proposes a 4-step AI-assisted peer review workflow — paulnovosad · 2026-11-09
- CoRL 2026 to host 'Do Robots Need World Models?' debate workshop in Austin — YunzhuLiYZ · 2026-11-09