RL never uses uniform sampling: bias sources and the RFCL fix
KyleMorgenstein · x · 2026-11-19
Kyle Morgenstein explains that uniform sampling is never used in RL training—even vanilla RL—since trajectories are inherently biased by state and policy distributions. A very narrow reset buffer can indeed bias training, but approaches like RFCL can expand the distribution. The thread's earlier point: reset state choice biases the model, but good reset state generation is much easier than reward tuning.
More from Research
- New essay makes the case for traditional social science in studying AI agent interactions — RobbWiller · 2026-11-19
- If a model generates valid math blackboards, does it understand math? — francoisfleuret · 2026-11-19
- The Zvi on GTA 6's 99 Metacritic score: 'Welcome to 2026. We need better benchmarks.' — NathanpmYoung · 2026-11-19
- Stanford blood test ages 11 organs from 5,000 proteins; Helios launches 1,000+ biomarker home test — jgooten · 2026-11-19
- Kaggle lays out roadmap to fix AI benchmarks: relevance, saturation, verification, infrastructure — MeganRisdal · 2026-11-19
- Researcher: my real peer review is asking the world's top 2-3 experts, conference reviews are noise — taiyasaki · 2026-11-19