BaRe-Mem: Bayesian Reliability Memory for Robust and Adaptive Agent Consultation
Peilin Feng, Zhengyang Huang, Soujanya Poria
cs.AI
2026-09-29
BaRe-Mem estimates advisor reliability from hidden states and online Bayes. On nine benchmarks and six models, it stays 1.5-2.8 points above solo even at 100% misleading advice.
Asking other agents can make you worse. Advisor skill is task-dependent, and a fluent wrong answer can talk a central model out of a correct one. Majority vote fails when advisors lock onto the same plausible error. Debate has the same failure mode: on the harder suite, once more than half the advice is misleading, both question-plus-peers and two-round debate fall below answering alone.
Text memories keep trajectories and skills. Latent memories compress experience into a state. Neither turns history into a per-question estimate of whether this external evidence is trustworthy, or whether you should listen at all. When the central model is already strong, collaboration can shrink or go negative. BaRe-Mem, from NTU's DeCLaRe Lab and Peking University, stores that estimate as an online Bayesian memory.
The central model stays frozen. No extra trained parameters. On each question it first answers by itself, then reads K advisor replies. Each candidate is encoded into a hidden state, split into a shared question belief ψq and an answer-content belief ψc. Reliability is a sum of three terms: how this source has done on similar questions, how the content itself looks, and a bias. A model that is solid at math should not inherit that score on code.
Correctness is coded as +1/-1. A Bayesian linear regression keeps a posterior over weights. A fixed step would treat every verified outcome equally; the Kalman gain applies a rank-one update whose size tracks current uncertainty, so history is not replayed. With no evidence, every source sits at 1/2. The output p is the predicted probability of being correct.
Attention patching is the interface because p is already a per-response scalar. It can rescale influence without another trained module or a retrieval prompt. p is used twice.
The same scores rank workers in a team: try the most trusted first, then the next if the report is rejected.
Nine benchmarks, two regimes. Supported: GSM8K, APPS, SQuAD. Challenging: PIQA, MMLU, OpenBookQA, SciQ, BBH, SuperGLUE. Six central models, six advisors. Misleading ratio from 0% to 100%: replies stay fluent and on-topic, final answers are swapped to wrong.
On the challenging suite, majority vote collapses first. At 100% misleading, advisor-only majority vote is 1.2; adding the central model's own answer still leaves Qwen3-14B at 2.8. An always-consult ablation that only reweights by reliability also drops below solo at 100% misleading. BaRe-Mem cuts the consult rate from about 85%–90% to 9%–21% and stays above solo for all six models at every tested corruption level.
| Central model | Solo | 0% misleading | 100% misleading | Consult rate at 100% |
| Qwen3-14B | 66.5 | 76.1 | 68.6 | 21% |
| Phi-4 | 63.4 | 71.2 | 65.0 | 18% |
| Ministral-8B | 58.6 | 70.8 | 60.6 | 9% |
Gains over solo: +7.2 to +12.4 points on the challenging suite, +4.3 to +13.7 on the supported suite. At 100% misleading on the hard suite, BaRe-Mem beats the always-consult ablation by 13.0 to 34.6 points and solo by 1.5 to 2.8. On the supported suite the consult rate barely moves, 87% to 85% for Qwen3-14B, because the advisor pool is still useful.
Predicted consultation gain tracks realized gain after verification, and the curves cross near zero. Writing outcomes for under 1% of questions, about 170 labels, already beats zero feedback; returns flatten after that.
On MuSiQue (2,417 tasks, 6,404 sub-tasks) the same memory routes workers instead of reweighting replies. Random routing with no check is about 23%. With a dataset verifier, Qwen3-14B and Phi-4 as leads reach 54.3% and 54.2%. Capable workers get tried earlier. The ceiling, at least one worker correct, is about 70%.
This is an attention bias plus an online regressor, no finetuning. You need hidden states, a way to patch softmax, and correctness labels after the fact. That matches pools whose quality drifts and tasks where asking is not always a good deal.
Whom to trust and whether to ask are separate knobs. Relative reweighting cannot save you when the whole pool is bad. Sparse gold labels are enough.
On the easy suite the lift is smaller, and when advice is clean the always-consult ablation often wins by a few tenths. The case for this method is contaminated advisors and uneven tasks, not a blanket replacement for debate.
The main text has no Limitations section. The appendix reports a calibration floor: at 100% misleading, empirical accuracy is near 0 while p stays around 0.2, approaching Φ(-1)≈0.159 because observation noise is fixed at variance 1. A linear Gaussian model leaves a tail on bad sources.
Updates need verified correctness. The experiments use gold labels; noisy labels are untested. Misleading replies are controlled answer swaps, not natural attacks or collusion. Black-box APIs that hide hidden states and attention cannot run the steering.
The team experiment uses dataset-provided sub-tasks, so it scores routing, not decomposition. Fine-grained numbers for the no-check and lead-agent-check settings should be read off the figure; the comparison that is clear is about 54% with a dataset verifier versus about 23% random.