Memory heuristics hurt LLM group decisions: full transcripts cut accuracy too
Anouar Imel · hf · 2026-10-28
A study from four Chinese universities asks what an LLM group's compressed memory should keep for a final decision maker. Two natural heuristics—keeping rarest claims or decision-relevant claims—both fail on hidden-profile tasks.
The authors introduce CrossProfile, a diagnostic benchmark of 60 templates with verified answer keys crossing supporting, decoy, and irrelevant facts (480 tasks). Across Qwen2.5-7B, Llama-3.1-8B and two DeepSeek models:
- Relevance selection predictably loses decoys—even a perfect relevance filter loses 23-31 points vs rarest-first in the classic hidden-profile setup;
- Rarest-first is no better than random;
- Due to social-proof bias and decoy inclusion, even full transcripts reduce accuracy vs no notes.
What matters is the balance of retained evidence, not summary accuracy: keeping a favorable fact while dropping an unfavorable one can flip the decision without any false claim.
More from Research
- 2020 physics paper links learning rate, Hessian and gradient noise via fluctuation-dissipation theorem — burny_tech · 2026-10-28
- EMNLP 2026 Hosts Tutorial on Failure Modes in Agentic AI, Mapping Four Stages of Agent Building — wzenus · 2026-10-28
- Jeremy Avigad: the AI debate belongs at the center with us, not the AI — Machine Learning Street Talk · 2026-10-28
- Meituan LongCat's FVQ makes discrete visual tokens match continuous representations — meituan-longcat · 2026-10-28
- Single neuron steering works far worse than contrastive-pair control vectors in practice — burny_tech · 2026-10-28
- Effective learning rate explains peak-then-decay schedules as bias-variance optimum — burny_tech · 2026-10-28