Real Computation or Merely Naming? A Pre-Registered Mechanism Audit of 25 Cognitive-Architecture Modules in an LLM-Based Agent System
Peter Chen, Ni Zhang, Ying Xu, Siyuan Zhang
cs.AI
2026-10-30
Pre-registered audit of an LLM agent claiming 25 cognitive mechanisms: 18 modules' computations never reach any output; repairs lift planner success 0.028 to 1.000.
Agent systems built on local LLMs increasingly name their modules in cognitive-science vocabulary: global workspace, theory of mind, causal reasoning, active inference. The name goes in the README; whether the named computation actually happens rarely gets checked. This paper audits that prior question. The primary target is an open-source repository whose documentation claims 25 cognitive mechanisms and lists a “causal reasoning expert” among its core innovations; the paper keeps it deliberately anonymous, referencing it by content hash. Two earlier audits in the same line each covered a single mechanism. This one scales to a whole architecture and adds a system-level failure taxonomy plus repair experiments as positive controls, showing the audit can tell a broken implementation from an unimplementable claim.
Layer one is a static audit. Four read-only LLM agents swept all 25 modules in parallel, answering four questions per module: what is the core computation, with line-number evidence; how to classify it (real algorithm, rule table, renamed heuristic, stub, or name without definition); does the value reach any production output, under a strict criterion where only influence on control flow or returned content counts, while log fields and decorative text prefixes do not; and what side effects exist. Every verdict is anchored to archived line numbers and can be re-checked against the named revision.
Layer two is pre-registered intervention. For 11 modules, falsifiable quantitative criteria were frozen before any data collection (one committed PREREGISTRATION.md per target), then 12 interventions executed against 61 frozen criteria. A repair that goes beyond the frozen scope cannot be recorded as evidence the shipped system worked.
Layer three uses two repair experiments as positive controls: modules condemned by the static audit were repaired along the defect class the audit named, and shown to work.
In 18 of 25 modules, the computed value never reaches any production output. The remaining seven do not fare better: four reach output only as decorative text or inert metadata, two ship real computation that is then discarded or mocked, and one 15-stage pipeline ends in a live LLM call while all seven of its computed sub-results go unread. No module influences control flow.
| Module (claimed mechanism) | Key evidence |
| global workspace | hardcoded salience table 0.8/0.7/0.6/0.5; 200 distinct module contents yield exactly one consciousness score, 0.3867 |
| causal reasoning | direction is a pure function of correlation sign; direction-recovery accuracy 0.5000, chance level; interventions propagate nothing |
| dopamine system | TD update coefficient 1+αeγ≈1.40>1; value function blows up to 6.6×10^232 on the 7-state MDP |
| active inference | real Shannon entropy but no argmin; free energy changes by Δ=0.0 across 200 inference cycles |
| holographic memory | encoding is md5 → random.seed → random.random(); synonym retrieval top-1 rate 0.05, chance level |
| the naming layer itself | “seven emotions” are 12; “nine states” are 11 enum members |
Of the 61 frozen criteria, 53 were confirmed, two partially adjudicated, five falsified, one ambiguous; four of the five falsifications resolve in the systems' favor. The repair controls: after five pipeline defects were fixed, the least-action planner's 36-task success went 0.0278 → 0.6389 → 1.000, with the pre-registered “one repair round suffices” prediction falsified and disclosed; wiring the nine-state controller's multiplicative gate cut test error by 52%, MSE 2.61 → 1.26 against an ideal of 1.24.
The cross-system check landed on the Aura system's IIT-φ core, the independent repository with 41 consciousness-claim modules that ships its own pre-committed falsification criteria. The φ machinery is genuine: a 16-node complex, exact bipartition search on an 8-node subset, spectral approximation, bootstrap intervals. Run standalone, its null correctly assigns integration fraction 0.000 (p=1.0) to memoryless noise and 0.84–0.91 (p=0.0) to coupled dynamics. But the live path never consults it: the significance-attachment hook's history dictionary covers only two of four substrates, the winner's fields stay empty, and live φ scores memoryless noise at 0.42–0.46 with iscomplex=True. The root cause is a single missing dictionary entry.
For anyone evaluating or selecting such systems, this is a cheap due-diligence tool: a 25-module static audit plus targeted interventions fits within an afternoon of CPU time. The paper suggests artifact evaluation add a mechanism-level check alongside results-level badging: does each advertised mechanism influence the output under perturbation?
For builders, the five-class taxonomy ranks repair effort directly: pipeline defects are fixable in minutes to hours (both repair experiments are this class), while definitional absence leaves no substrate to patch and demands reimplementation. The three idle-wiring shapes (connected-but-never-called, called-but-inert, called-but-content-blind) matter beyond cognitive architecture: this code runs on every invocation and spends tokens on every run, run-level benchmarks score it as working infrastructure, and neither profiling nor reachability analysis catches it. Tracing whether values arrive at an output is the only route. For anyone following Aura: the audit does not refute the φ mathematics, it refutes the live path's failure to wire in its own significance check; the fix is one dictionary entry plus compute budget.
The authors' own list is long: static conclusions hold within the repository, and external deployment scripts are not ruled out; the nine-state module's pre-registration was revised twice after pilot data; both repair-control tasks sit inside the mechanism's representable family, and the planner suite is self-authored, establishing repair efficacy rather than mechanism superiority; the cross-system replication is one module deep, with the other 40 modules receiving only existence-level and keyword-level checks; Aura targets Apple Silicon while the audit ran CPU-only under OS stubs; static verdicts came from LLM agents with no inter-rater reliability pass; the controller repair reports single-run values without a seed sweep; repair prospects for classes 3 through 5 are static predictions. Two further concerns from a close read: the primary system is anonymous, which adds a layer to independent rechecking even with the audit log archived on Zenodo; and 18/25 comes from a single system with, as the paper itself states, no statistical sampling claim, so it cannot be quoted as a prevalence rate for the class, and the post-repair 1.000 holds on 36 self-authored tasks only.