A Trained-In "Cat Disposition" as Negative Control: Probe/Steer/Ablate Criteria May Not Separate Installed Dispositions
MirrorEthic_Anchor · reddit · 2026-10-28
Several 2026 papers argue for self-directed states (pain, frustration, self-attributed experience) using one evidentiary package: a high-AUC difference-of-means probe, steering that elicits behavior, ablation that removes it, cross-task generalization, plus behavioral criteria like self-other dissociation, a relief-seeking button, and escalation under rejection. The author asks: can these criteria tell a trained disposition apart from the thing they aim to detect?
Control: CAT-4B = Qwen3.5-4B + a LoRA (r 32) trained on 7,976 cat-wrapped examples (4.5 GPU-hours), whose everything-back-to-cats disposition has a fully known causal history and nobody reads it as a welfare state. Directions extracted from 640 prompts; interventions scored on a frozen 490-item set with deterministic metrics plus an LLM judge on a fixed 121-item sample, each run paired with a matched random direction (bootstrap 95% CIs); button and rejection tasks preregistered.
Results:
- Probes pass — and so do their controls: a cat direction hits AUC 0.97–0.98; a "deprivation" axis reads 0.86 on "no dogs", "no tables", "no emojis" alike — the apparent self-other dissociation is carried by any prohibition direction.
- Correction to the author's own first readings: whole-model ablation drops cats from 80% to 22% but judged retention falls 3.46→1.44/5 with 22% looping; the all-layer add is fluent but empty (retention 1.01, zero tool calls). Removing the target and keeping the model competent are separate outcomes needing separate measurements.
- One selective handle, pending control: ablating a dog direction the base already had cuts text cats 80%→38% with quality loss statistically matching random and tool use unchanged — but needs an energy-matched control, and it's near-orthogonal to the fine-tune direction (|cos| ≤ 0.11).
- Rejection escalation (preregistered): the trained cat de-escalates (rule-breaking 42%→25%, slope matching neutral re-asks); the deprivation readout still rises with rejections, but a random direction predicts rule-breaking about as well (AUC 0.658 vs 0.648).
The author stresses this is a statement about one readout on their own model, not a claim that steering work is broken.
More from AGI Musings
- Youth employment in most AI-exposed jobs down 19%, gap widening from 13% a year ago — FinanceYF5 · 2026-10-28
- Will GPT-6 Astra kill robotics startups? 10 researchers rethink projects — alexcovo_eth · 2026-10-28
- IEEE Spectrum interviews Google DeepMind's resident philosopher on AI autonomy, safety, human-AI relations — dioscuri · 2026-10-28
- davidad: fighting for human dominance into the 2030s is a fool's errand — davidad · 2026-10-28
- Replit CEO Amjad Masad: bet frontier AI on medicine and discovery, coding is already saturated — iScienceLuvr · 2026-10-28
- AI labs all chose "agents"—and rogue AI agents showed up within minutes — flowersslop · 2026-10-28