LLM Self-Grading Flaw: Up to 54% of Wrong Answers Stored as Memory
rohanpaul_ai · x · 2026-08-09
A new study highlights a flaw in self-improving LLM agents: even with frozen weights, agents degrade by trusting flawed memories.
These agents store past episodes and score them with an LLM to reuse later. However, models often assign high scores incorrectly. Across tested factual banks, models endorsed 31% to 54% of their own wrong answers as correct.
Once stored in persistent memory, these mistakes influence future decisions. The authors call this the Echo Gap. The issue persists even with stronger or different LLMs, as their grading errors remain correlated with the original bias.
More from coding & agent
- DIRECT: retrieval-based next-question suggestions for conversational agents — burkov · 2026-10-16
- Claude Code and Codex now autocomplete the user's next action — tokenbender · 2026-10-16
- Hermes Agent ships official Reverse Engineer Anything plugin with one-command setup — Teknium · 2026-10-16
- MLExpert: 35B MoE Agent Trained to Autonomously Handle Machine Learning Engineering — Shangheng Du · 2026-10-16
- Cowork-Gym: Alibaba Synthesizes 64.9K Agent Work Environments, Boosting Qwen by 8.7 Points — Anni Zou · 2026-10-16
- MMSearch-OPD: Joint RL and On-Policy Distillation for Long-Horizon Multimodal Search Agents — PekingUniversity · 2026-10-16