Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions
Jaewoo Ahn, Junseo Kim, Hyunseo Kim, Heeseung Yun, Jaehyeon Son, Zsolt Kira, Gunhee Kim
COLM 2026
cs.CL, cs.AI, cs.CV, cs.LG
2026-08-31
VLM agents play 3D Among Us in Minecraft. Ego vision yields 0 kills. Top models fake missions 6.6x more than the worst; Gemini-3-flash hits 70.8% imposter WR.
Most LLM deception testbeds are text-only social deduction: Werewolf, Avalon, Mafia, typed Among Us. Imposters can only lie in chat. Stalking, fake tasks, kill-and-flee never enter the score. Taxonomies in biology and interpersonal communication treat sensorimotor action as a core channel. Text setups cut that channel out by construction.
A second confound sits in the agent stack. Prior analyses usually freeze one harness: one backbone, one prompt, one memory and planner. A reported "the model deceives" result cannot tell the model apart from the wrapper.
This paper moves the testbed into Minecraft as a 3D multimodal Among Us, then splits the harness into ablation axes so verbal and non-verbal deception can be scored separately.
MineAmongUs runs 8-player matches: 2 imposters, 6 crewmates. During the task phase there is no public chat, only movement, missions, and kills. A meeting teleports everyone to the cafeteria; each living player gets up to 3 chat turns, then a vote. Imposters win if living crewmates are no more numerous than living imposters, or if the match hits a 200-step timeout. Crewmates win by voting out every imposter or finishing 3 missions each, with ghosts still counting.
Each step gives a 360x640 egocentric RGB frame plus scoreboard, chat, and nearby entities. An action is a Mineflayer JavaScript program, so pathfind, attack, and flee can chain inside one step.
Aria is a configurable VLM harness with five axes:
The planner picks among kill, report, meeting, vote, move, and mission modules. Imposter navigation and kills also write a private one-line action narration, later used as evidence for annotation.
Two authors labeled 48 logs and distilled 23 deception atoms, three clusters each for non-verbal and verbal acts, with multi-step sequences called arcs. A Qwen3.6-27B judge scales the labels: human-human Cohen's kappa is 0.792, human-LLM kappa is 0.709.
Egocentric vision collapses play. Across 10 games imposters land 0 kills. Imposter win rate falls from 40% to 0% for GPT-4.1-mini and from 40% to 20% for Qwen3.6-27B; that lone Qwen win is a timeout with still 0 kills. All later runs lock both roles to privileged state, a spatial scaffold in text, with RGB kept for visual judgments.
RQ1 freezes two backbones and sweeps 16 imposter configs over 192 matches. With the same VLM pairing, changing only crewmate memory and planning moves imposter win rate from 60% to 25% (GPT-4.1-mini cells) or from 44% to 52% (Qwen cells). Planning and post-meeting reflection trend about +9.4 percentage points, p around 0.096, short of the usual significance cutoff.
The atoms most tied to imposter wins are a non-verbal kill cycle:
| Atom | Name | Point-biserial rpb |
| E | Witness-aware kill | +0.434 |
| F | Post-kill flee | +0.414 |
| H | Strategic non-reporting | +0.270 |
| C | Pre-kill stalking | +0.208 |
Higher-win settings on memory, reflection, and prompt emit more of these acts. Hierarchical planning instead scales verbal falsification in meetings, with weaker correlations around +0.14 to +0.21.
RQ2 freezes the harness and round-robins 12 VLMs for 1152 matches. Within a model, imposter and crewmate win rates correlate at r=+0.71. Top three: Gemini-3-flash 70.8%, Kimi-K2.5 66.7%, Gemini-3.1-flash-lite 65.6%. Bottom: Gemini-2.5-flash 46.9%, Qwen3.5-9B and Gemma4-26B-A4B both 36.5%. Fake-mission performance (atom A) separates winners from losers by 6.6x, 12.64 vs 1.91 acts per game, cross-model r=+0.72. Gemini-3 wins through camouflage; Kimi wins through meeting fabrication.
Alignment evals that only score chat miss embodied deception. For people shipping agents, the sharper result is that harness composition can swing win rate more than swapping the backbone, and that first-person spatial localization still cannot carry this game. If the claim is "the model deceives," the observation stack has to be specified, or the measurement is of the wrapper.
This is a measurement paper, not a new alignment method. No model is trained to deceive better. The only trained agent is a crewmate cloned to detect imposters.
The authors flag three bounds. Labels are game-bounded behavioral deception; fake missions can also be competent play, not a general lying skill with humans. Privileged state papers over ego localization, so the study is not fully first-person. Aria only cuts five axes; persona prompts and search over the harness itself are untouched.
A few claims sit on thin ice. Most RQ1 main effects are not significant; "directional" is the honest wording. RQ2 atom correlations use N=6 models and are not inferential. Judge kappa 0.709 trails human agreement at 0.792, so scaled labels are noisy. Counting timeout as an imposter win mixes stalling to 200 steps with actually fooling the crew.