More accurate behavioral predictions with hybrid Bayesian-connectionist models
Brenden M. Lake, Akshay K. Jagadish, Guangyuan Jiang
cs.LG
2026-08-23
BBT distills a Bayesian model into a 30M transformer then fine-tunes on humans, beating Bayesian and both ablations on four concept-learning tasks.
Cognitive modelers have spent decades choosing a side. Bayesian models make it easy to write down structured priors: rules, grammars, programs, and then to test those commitments against few-shot human judgments. The cost is rigidity. Hypothesis spaces and parametric forms are frozen in, so the shortcuts in real behavior have nowhere to go. Computing the posterior predictive often means summing over a huge hypothesis space, which is intractable.
Neural networks flip the tradeoff. They are flexible and cheap at inference, one forward pass. They are also hungry for data, and their inductive biases are hard to inspect or swap. Human experiments typically offer a few hundred participants, not the scale a 30-million-parameter transformer wants.
Lake, Jagadish, and Jiang stitch the two toolkits together. Distill a Bayesian model into a network from synthetic data, then fine-tune that network on human responses. They call the recipe Bayesian distillation with Behavioral Tuning (BBT). It is a model-fitting procedure, not a claim about how children develop.
Each training episode is a mini learning problem: a support set of input-output pairs plus a query. The network predicts the human answer. The backbone is a 30-million-parameter encoder-decoder transformer, 4 layers on each side, 8 heads, 512-d embeddings, shared across all four case studies.
Stage 1, distillation, uses only synthetic data. Sample a hypothesis from the prior, then sample observations from the likelihood. The network learns to approximate the Bayesian posterior predictive: the distribution over the next label given the support set. Forward sampling is enough; no posterior inference is required. On the number game, the distilled network matches the original Bayesian model at Pearson r = 0.997 across 5,100 held-out queries.
Stage 2 fine-tunes the same weights on human behavior and keeps the checkpoint with the best validation loss. Fine-tuning is not bound by the original prior, the original likelihood, or even Bayesian updating. If people use simpler rules, one-to-one mappings, or hypotheses the Bayesian model never listed, the network can follow. The ablations are distillation-only, fine-tuning-only, and both stages.
Held-out log-likelihood of human responses (higher is better):
| Model | Number game | Logical concepts | Shepard | Compositional |
| Bayesian | -31332.6 | -79023.3 | -5554.4 | not built |
| Distillation only | -31435.7 | -77249.9 | -5601.1 | -466.4 |
| Fine-tuning only | -31465.2 | -106926.3 | -5924.7 | -1653.6 |
| BBT | -28157.9 | -69423.8 | -5260.8 | -343.6 |
No separate Bayesian inference engine was built for compositional instructions; the distilled network is the approximation. Fine-tuning-only finishes last in every column. A 30M model cannot be trained from this much human data without the synthetic prior.
On the number game, BBT matches the human yes-rate at r = 0.81 and RMSE 0.153, against 0.66 and 0.302 for the original Bayesian model. Fine-tuning the Bayesian model itself does not close the gap: a free prior over 5,084 hypotheses reaches r = 0.67; freeing the likelihood as well (about 519k parameters) reaches r = 0.78, still 578.8 nats behind BBT. For the support set {66, 78}, people and BBT treat "evens" as the main story at weight 0.46; the Bayesian model puts it at 0.05 and prefers "multiples of 6" at 0.53.
On Piantadosi's logical concepts, BBT beats the best of 21 Bayesian models by at least 7,826 nats. On the last 8 object sets, r goes from 0.87 (distillation-only) to 0.88, RMSE from 0.214 to 0.190. Humans are 78% correct; both networks are 69% correct on the ground-truth rule. The extra fit is not better rule recovery. People fall back to simpler feature hypotheses, and after fine-tuning the best-matching hypotheses have 2.2 fewer function calls. On 11 concepts never seen in fine-tuning, the second stage still lifts the pretrained network by at least 481.7 nats.
The Shepard task adds memory load: previous trials are not left on screen. BBT learns extra embeddings for block number and age group. Learning curves correlate with humans at r = 0.938; fine-tuning-only is essentially flat (r = 0.022). Simulated accuracy is 86.9% for younger adults and 77.7% for older adults, versus 78.6% and 65.9% in the data: the age gap is there, the absolute numbers run high. A model fine-tuned on Lewandowsky et al. predicts held-out Badham participants collected six years later at -5,331.1, close to a model fine-tuned on Badham's own train split at -5,298.7. By the end of block 1, the share of Type I (single-feature) hypotheses rises from 0.02 after distillation to 0.17 for younger and 0.05 for older.
Compositional instruction learning asks people to acquire several rewrite rules at once. In the original experiment, 80.7% of answers from 23 participants matched the designed symbolic system. This paper adds 166 participants across 8 rule systems. BBT gains 122.8 nats over distillation-only and slightly beats a previous model with hand-engineered biases (-343.6 vs -349.2). Of human errors, 24.4% are one-to-one mappings and 23.3% iconic concatenation; BBT shows 44.4% and 7.0%, distillation-only 10.4% and 0%. The directions match, the rates do not.
For cognitive modelers, this is a reusable recipe: write the prior in Bayesian language, predict with a forward pass, and let fine-tuning absorb the residual human structure. A 30M transformer can be trained on a little over a hundred people if synthetic data has already loaded the logic of the hypothesis space. The four tasks sit on the same line: infer a rule from a handful of examples. Fine-tuning alone collapses. Distillation alone misses the heuristics.
Relative to prompting or fine-tuning large language models as cognitive models, BBT's training data is controlled. Distillation teaches the task; the context window does not dump earlier human response sequences. That makes the successes easier to interpret, at the cost of being task-specific and empty of world knowledge. This is a methods increment, not a new theory of mind.
The authors flag several. The two-stage recipe is not a developmental story; do not read it as children starting near Bayesian solutions and then drifting. Reading out a fine-tuned network depends on which layer, which episodes, and which variables the analyst picks. Sparse approximation can only reweight hypotheses from the original Bayesian model, so genuinely new hypotheses will be missed. Informal pilots suggested broader priors and cleaner likelihoods distill better; that comparison is not systematic.
The numbers leak in other places. BBT never picks up the human preference for the entire 80s range on one number-game item. On Shepard, model accuracy sits systematically above people. On instructions, one-to-one bias is overstated and iconic concatenation is understated. All four tasks are laboratory concept learning, far from natural categories. Source code is said to live on GitHub; the paper text does not give a repository URL.