Time-Aware RL Revives Sutton's 30-Year-Old Semi-MDP Framework for LLM Agents
sherryyangML · x · 2026-10-16
Researcher sherryyangML presents new work inspired by Sutton, Precup & Singh's classic "Between MDPs and Semi-MDPs," applying semi-MDP concepts to modern agentic RL for LLMs.
- Key result: On NanoGPT, reward-rate RL reduces pretraining loss faster than vanilla RL.
- Motivation: With autoresearch and MLE agents drawing interest from Karpathy, Quoc Le, and Jeff Dean, agents must learn to weigh an experiment's payoff against a limited research budget — choosing the next experiment is deciding how to spend time.
- Takeaway: RL foundations remain vital; time-aware RL could be an important ingredient for agentic RL.
Related event: Reward-Rate Policy Gradient: Time-Aware RL Speeds Up Agent Training(3 posts)→
More from AGI Musings
- AI-driven job losses could turn $10T consumer debt into a $1T banking hole — churchkey · 2026-10-16
- Into the Unknown: an AI agent benchmark of 15 open research problems from US national lab scientists — Jeande_d · 2026-10-16
- Soon your AI will meet their AI — and humans without agents can't compete — danfaggella · 2026-10-16
- Analyst: in a year, models 2-3x better across the board at 10x lower cost, token use up 10-20x — DeryaTR_ · 2026-10-16
- AI will make many discoveries for us, but adversarial domains like soccer will thrive — natanielruizg · 2026-10-16
- MIT's Buehler on AI as a new language of science: superintelligent partners and executable papers — CatAstro_Piyush · 2026-10-16