Base Labs investigates why models forget facts they iteratively learn, with 13B talkie
baseten · x · 2026-10-13
Base Labs' weekly Rollout asks why models forget facts they iteratively learn. Extending their NeurIPS workshop submission, they use talkie, a 13B model trained only on pre-1931 text. Experiments show sequentially writing 20 facts causes earlier facts to be overwritten: recall is near 100% right after a fact's own write but drops sharply after all 20, with mid-sequence positions (6–15) hit hardest while earliest and latest positions survive better.
More from Research
- DUET paper co-evolves agent and LLM grader, hits 96.5% on Finance Agent benchmark — dair_ai · 2026-10-13
- Robot deployment is becoming the operating layer for post-training physical AI policies — paigeinsf · 2026-10-13
- New J++ Lens interpretability method delivers 55% more faithful readouts than J-Lens — wesg52 · 2026-10-13
- Spatial biology slide costs stuck at $5-10k for five years, defying Moore's law — anshulkundaje · 2026-10-13
- Google Research trains user simulators with latent intrinsic feedback, cutting RL compute by over half — cocoweixu · 2026-10-13
- Helmholtz-SDE closes the approximation gap in simulation-free latent SDEs — scott_linderman · 2026-10-13