AI2 researcher teases a RULER-style long-context benchmark isolating state-tracking complexity
lambdaviking · x · 2026-11-04
In a thread clarifying an earlier discussion, lambdaviking explains the team is building a RULER-like long-context evaluation focused on state tracking: deterministic synthetic tasks solved over the input without reasoning tokens. He notes some RULER sub-tasks (niah-1, VT) already involve state tracking, but their benchmark aims to cleanly isolate different levels of state tracking by theoretical complexity.
More from Research
- Agents will soon edit their own weights and memory — but who studies when to use which? — sarahcat21 · 2026-11-04
- Claude fixes OCR doom loop by editing 0.0085% of weights, no training, $10 — huggingface · 2026-11-04
- Hand-writing all weights of a 100k-param model beats training it at spotting fake legal citations — AaronBergman18 · 2026-11-04
- Gradient-Sealed Unlearning aims to stop fine-tuning from restoring harmful model capabilities — burkov · 2026-11-04
- New method forecasts GSM8K accuracy within 3 points using 5600x less compute — kenziyuliu · 2026-11-04
- Team burns 10B tokens in 2.5 days to build Omniscience, a city-scale world model — Kuprel · 2026-11-04