Carmack digs into RL Q-value overestimation, finds elegant fix in Relative Value Learning
ID_AA_Carmack · x · 2026-09-18
- John Carmack shares his exploration of a Relative Value Learning paper, noting he has often observed estimated Q values exceeding observed returns in RL — sometimes substantially — which hurts performance via bootstrapping.
- After wrestling with the paper's "Banach space of bounded antisymmetric pairwise functions" framing, he realized the machinery essentially amounts to subtracting two value functions.
- With an LLM's help he simplified it into an elegant form: subtracting the mean TD error from individual sample TD errors retains all the benefits while removing the offset.
- He also published the full exploratory chat log, itself an interesting case study in LLM-assisted paper understanding.
More from Research
- alphaXiv launches "Musolesi number" to measure your academic coauthor distance — mircomusolesi · 2026-09-18
- Anthropic quietly builds Bay Area wet lab to let Claude direct biology experiments — mkheck · 2026-09-18
- Opportunistic AI Imaging Opens a New Window Into the Kidney via Routine Echocardiograms — EricTopol · 2026-09-18
- arXiv One-Strike Policy Could Mean a Lifetime STOC Ban, Researcher Warns — Aaroth · 2026-09-18
- RSI-Exam updates: GPT-6-astra holds #1 at 0.5126, Anthropic's Fable 5.1 debuts at #2 — HuaxiuYaoML · 2026-09-18
- Entropy doesn't track what matters in chain-of-thought, BlackBoxNLP paper finds — gsarti_ · 2026-09-18