Exploring Collaboration between a language and a non-language agent
Harini S, Somesh Singh, Yaman K Singla, Rajiv Ratn Shah, David Doermann, Balaji Krishnamurthy
EMNLP 2026
cs.CL, cs.AI
2026-09-01
Adobe projects Lc0 hidden states into 32 continuous tokens inside Qwen3. One 14B model matches or beats task specialists and GPT-5 with tools on six chess tasks; the matched text interface trails by 2-3x after DAPO.
LLMs now sit on top of specialist subagents. When both sides speak language, a text handoff is fine. In chess, robotics, and driving, the strongest specialists are not language models. Their skill lives in continuous hidden states: policy mass, value estimates, look-ahead structure. Current pipelines squeeze that into a few tokens of prose, such as "Nc2, P=0.34, +0.12". Adobe MDSR calls the leftover loss verbalization debt.
Chess is a clean testbed. Engines already beat grandmasters, but they cannot commentate, prepare against a style, or say whether a puzzle is interesting. LLMs have read more chess prose than most experts, and still play poorly. Tasks that neither side can do alone measure how much the interface throws away.
Latent state internalization projects the subagent's continuous state into the LLM token stream instead of translating it first.
The frozen specialist is Lc0-BT4, a 15-layer Transformer (240M parameters, 1024-d state). The backbone is Qwen3 at 4B, 8B, and 14B. A three-layer MLP named LatentBridge maps the engine's penultimate-layer activations into k=32 continuous tokens whose width matches the LLM hidden size. k=32 is where a projector-and-policy sweep saturates. Layer 14 of 15 is used because Stage-1 alignment loss is lowest there, and prior interpretability work already found value, square, and look-ahead features in that block.
A trace interleaves three token types: language (chain of thought plus the engine's text summary), actions that advance the board, and the 32 state tokens. The LLM decides when to call getpolicy, on the current position or on a counterfactual reached by a candidate move. Both LLAMIA and the matched control receive the same text serialization. Only LLAMIA also gets the latent tokens.
Training has two stages. Stage 1 freezes the LLM and aligns the projector on engine self-play state-policy pairs, so language skill is not overwritten. Stage 2 jointly trains the LLM and projector with DAPO, a group-relative policy optimizer with asymmetric clipping. State-token positions are gradient-masked. LLAMIA-Verb is the same recipe with the latent channel removed.
LLAMIA-Bench has six tasks: behavior cloning on the Maia distribution plus three OOD Wild splits, puzzle difficulty, puzzle interest, move annotation, and full-game commentary. A single LLAMIA-14B leads every column.
| System | Maia move-match | Wild | Diff. ρ | Interest ρ | Rationale BLEU-2 | Commentary G-eval |
| GPT-5+Lc0 text tool | 45 | 40 | 0.48 | 0.10 | 37.5 | 0.55 |
| LLAMIA-Verb-14B | 45 | 39 | 0.45 | 0.08 | 33.2 | 0.40 |
| LLAMIA-14B | 53 | 49 | 0.71 | 0.52 | 45.8 | 0.75 |
Maia and Allie train on tens of millions of games. LLAMIA uses 20K. On in-distribution Maia buckets it sits in the expert band (53 vs Maia 52 and Allie-Policy 54). On Wild it beats the strongest specialist by 4 points (49 vs 45).
Interest is the diagnostic. Every verbalized system stays at Spearman ρ≤0.12. LLAMIA-14B reaches 0.52. The 4B latent model already scores 0.38; the 14B verbal model is still at 0.08. Filling the same 32 slots with a ChessCLIP board encoder, rather than Lc0 state, drops interest to 0.08. Shuffling LLAMIA's own latent tokens collapses toward Verb.
The debt grows through DAPO and ends at about 2-3x. Scaling the LLM from 4B to 14B lifts both curves and does not close the gap. The latent model also changes strategy by task: engine-follow for play, consult-then-override for cloning, counterfactual queries for commentary. Verb collapses to engine-follow on every row (62-76%). Call count is lower (1.9 vs 2.9 at 14B), so extra 32 tokens per call do not raise latency.
On Go with KataGo-b18, 8k positions get LLAMIA-Go-14B to 48/50 top-1 human match at ranks 5k/5d, about 10 points above the verbal control. In a 12-player study (all rated ≥1700), LLAMIA passes as human in 61% of games; Verb is detected 72% of the time. Commentary preference is 72.2% for LLAMIA, driven by insight rather than factual accuracy.
For anyone wiring LLMs to non-language specialists, the claim is structural: a text tool API discards task-relevant signal whenever the answer lives in a full policy distribution or in multi-step state. Bigger backbones and more RL budget do not buy it back in the range they tested.
The practical catch is weight access. Closed models cannot ingest these tokens. The paper's workaround is to let an open LLAMIA speak language to the closed model while it internalizes the specialist. Transfer past board games is unshown. The usable takeaway is narrower and still sharp: if the specialist's value is in its hidden state, treat the interface like a multimodal projector, not like ReAct.
The main text has no standalone limitations section. Closed-weight models cannot use the channel. Go is behavior cloning only. The human study is 12 players. Interest at ρ=0.52 is far from a production ranker. Strategy labels and commentary scores both use GPT-4o. Lc0 and KataGo are clean, open, rule-based engines; whether robot or driving latents align the same way is untested. The abstract names GPT-5.1; tables report GPT-5. Numbers above follow the tables.