Bringing Comparative Cognition To Computers
Konstantinos Voudouris, Lucy G. Cheke, Eric Schulz
cs.AI
2025-03-05
Human psych tests on LLMs miss real skills and invent fake ones. Helmholtz and Cambridge say borrow a century of comparative-cognition controls.
Psychological tests are now routinely run on LLMs and reinforcement learning agents: exploration, risky choice, reasoning about uncertainty. Two motives sit behind this. One is practical, map what the systems can and cannot do so they can be improved and used safely. The other is scientific: AI may be a new class of cognitive system, worth studying both on its own terms and as a mirror for human minds.
Those tests were built for people. Applied to subjects with different architectures, training histories, and sensory interfaces, they fail in two directions at once. Real capacities get missed. Fake capacities get awarded. Comparative cognition, the study of animal minds, has fought this measurement problem for more than a century. This three-page position paper from Helmholtz Munich's Human-Centered AI group and the University of Cambridge (Psychology and the Centre for the Future of Intelligence) argues that those methods should now be used on computers.
The conceptual stake is what the authors call the cognition thesis: cognition can arise in structurally different systems if they are organized in the right way. Cognitive science, on this view, exists to specify that organization. The historical arc runs from primates, birds, and cetaceans, through bees, ants, and cephalopods, into bacteria, plants, and slime moulds. Computers and LLMs are the latest candidate. A large literature already describes LLM behavior with the same vocabulary used for people.
Sharing vocabulary is not a method. Comparative cognition supplies two experimental disciplines.
To avoid under-attribution, inspect the test before you inspect the subject. Domestic dogs and object permanence, the ability to track an object after it is hidden, is the textbook case. Early work used the invisible displacement task: a reward is placed in a movable container, the container is moved behind an occluder, the reward is left in the new location, and the empty container is shown. Dogs fail to find the new location. The first conclusion was that dogs lack object permanence. Later work showed the dogs were tracking the container. Independent measures now indicate they do have object permanence.
The AI analog is large-number arithmetic. Poor scores get read as missing a basic human capacity for numerical manipulation. A closer cause is often tokenization, the way LLMs chunk text into atomic symbols. Frequent numbers such as "100" and "99" become single tokens, so sums like "100100100 + 999999" fall apart. Singh and Strouse (2024) showed that when this tokenization artifact is reduced, arithmetic in frontier LLMs rises sharply.
To avoid over-attribution, rule out alternative explanations. Clever Hans, the early-1900s horse that appeared to do arithmetic, was reading unconscious cues from his owner. Deep networks can classify everyday objects at accuracies that sometimes beat humans. Show them adversarial images that a person can barely tell apart from the original, and accuracy collapses. Ilyas et al. (2019) argued these models lean on superficial features such as texture and pattern: a brittle, non-human form of object recognition. If a Clever Hans effect is left in the protocol, similarity gets sold as sameness.
The two disciplines have to be used together. Fixing under-attribution lowers the bar. A lower bar lets over-attribution in.
This is a position paper. There is no new experiment, no new benchmark, and no comparison table against a baseline. What it offers is a pair of misattribution types, each tied to published cases.
| Direction | Animal-cognition precedent | AI case | Mechanism named here |
| Under-attribution | Dogs fail invisible displacement | LLMs fail large-number arithmetic | Tokenization treats common numbers as atoms |
| Over-attribution | Clever Hans "does arithmetic" | Deep nets beat humans at object recognition | Adversarial examples expose texture-level features |
The paper does not report how many percentage points arithmetic improves after tokenization is controlled, nor how far accuracy falls under attack. Those numbers live in the cited papers. This text only says performance "greatly increases" and "unexpectedly collapses."
The framing citations are Hagendorff et al. (2023) on machine psychology and Herbert Simon (1980) calling cognitive science the newest science of the artificial. On whether rich psychological terms should be applied to AI, it points to Shevlin and Halina (2019).
For anyone running capability evals, this is a checklist, not a score. Before you write the item, ask whether the task is probing an interface defect (tokenization, prompt format, visual preprocessing) rather than the capacity you named. Ask whether a behavior that looks like a capacity is already explained by training data, prompt leakage, or surface statistics.
The same questions apply to safety and product reports. A finding that "the model cannot do arithmetic" is under-attribution if tokenization was never controlled. A finding that "the model has human-level object recognition" is over-attribution if adversarial examples were never run. Both send the next engineering cycle the wrong way.
There is no eval suite here. The executable claim stops at: control for test mismatch and alternative explanations before you announce what an AI can or cannot do.
The authors do not include a Limitations section. The gaps are still obvious.
There is no operational standard. The paper never says what counts as "taking into account the constraints of the system" or "adequately controlling for alternative explanations." No checklist, no decision tree, no set of counterexamples. A reader cannot replicate a "comparative-cognition-grade" AI experiment from this text.
The cases are illustrations, not a survey. Tokenization-vs-arithmetic and adversarial-vs-recognition get one citation each. The paper does not estimate how common either error is in current LLM evaluation. Contamination, in-distribution score-chasing, and benchmark leakage are AI-specific problems that barely exist in animal labs, and they are not treated here. Transplanting comparative cognition leaves those differences hanging.
The cognition thesis is a stance in this paper, not a hypothesis the paper can test. Jumping from slime moulds to Transformers still needs a falsifiable account of "appropriate organization." That work is deferred to cognitive science at large.
The document is a three-page arXiv preprint dated March 2025, with no venue acceptance listed. Treat it as a comment, not as new evidence.