Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents
Nan Li
EMNLP 2026
cs.CL, cs.LG
2026-08-28
On Qwen3.5-2B, broad SFT, Wordle turn-DPO and LoRA scale 0.85 lift public clemscore from 10.67 to 38.92; static stays 44.14 vs 44.24, OOD is 7.88.
Static benchmarks test single-turn answers. Dialogue games test a different skill: the model must carry state across turns, read the feedback it just received, and pick a legal action under tightening constraints. A guess rules out letters. A clue limits the partner. A move changes the room the next command is interpreted from.
The LM Playschool Challenge makes this a formal contest. The official Qwen3.5-2B baseline scores 10.67 public clemscore; the 27B sibling scores 60.30. Wordle transcripts from the 2B model show the gap is often not missing game knowledge. The format is right, then the next local decision is wrong: it repeats a rejected guess, emits the wrong length, or uses a letter the feedback just painted red. Those errors are frequent, rule-checkable, and pinned to the decision after feedback.
The training question is where to put supervision on the trajectory, not whether dialogue-game data helps.
Nan Li at Utrecht submitted playornotplay as a three-step recipe: Acquire, Repair, Preserve. All trainable stages use LoRA (r=16, α=32, all-linear, dropout 0.05) on Qwen3.5-2B in text-only mode, about 5.3 hours on one A100-80GB.
A control that preferred entire remaining dialogues over a single next action collapsed protocol: unfiltered whole-dialogue DPO scored 12.35 and started emitting natural-language preambles before game moves. Filtering the chosen side to stronger source models recovered only to 25.76; the main abort patterns stayed. The contrast has to land on the few post-feedback decision tokens.
The review manuscript described six stages. Continued weak-game SFT and a Wordle GRPO probe later proved to update zero weights: one froze the adapter, the other built an optimizer on an empty parameter set. Bug-fixed reruns netted -0.30, so the submitted sequence keeps only the four stages above.
The official final evaluation (snapshot 2dd5a533) is the headline. The pinned public-dev set has 14 games and 67 episodes; a few points of movement should be treated as noise.
| System | Public clem | Closed ID | Closed OOD | Static |
| Qwen3.5-2B baseline | 10.67 | 13.41 | 3.72 | 44.24 |
| playornotplay (2B) | 38.92 | 41.17 | 7.88 | 44.14 |
| Official 4B baseline | 26.66 | 34.02 | 17.99 | 51.33 |
| Official 9B baseline | 31.91 | 41.12 | 24.91 | 53.90 |
| DAIR SFT-only (same 2B) | 46.01 | 46.57 | 15.62 | 44.35 |
Public Played rises from 27.22% to 78.63%, Quality from 39.20 to 49.50. Closed in-domain rises on both axes: Played 47.32 to 82.62, Quality 28.33 to 49.83. Out-of-domain Played falls from 35.19 to 33.08, below the untrained base. The OOD clemscore move from 3.72 to 7.88 is conditional quality on fewer played episodes.
Of 84.86 total positive OOD movement, three held-out Wordle-crazy variants contribute 70.02 (82.5%). The largest hidden in-domain gains sit in games no later phase targeted: TMW GraphReasoning +71, ImageGame +60, AdventureGame +58. The Wordle family that received all repair data stays near zero on hidden in-domain episodes (3.3, 10.0, 3.3).
On the development trajectory, broad SFT is the largest single step (+30.80). Turn-DPO pass 1 is positive on both the submitted rail and the controlled single-host campaign (+2.98 and +1.49). Pass 2 and branch-DPO flip sign across those two rails. Aggregate static score is approximately preserved, with redistribution underneath: BBH Quality 0.00 to 30.47, IFEval 68.52 to 51.85, EQ-Bench 64.78 to 56.43.
For post-training a small interactive agent, this paper is a map of where supervision belongs. Most of the lift is broad imitation that teaches the model to finish an episode. Local failures that a rule can catch are a good fit for short preference pairs with a shared history. Treating the rest of the dialogue as a DPO sample, in a protocol-strict environment, wrecks the format. Scaling a LoRA delta is a cheap, training-free knob for pulling static score back from the specialization tax.
This is not the best 2B entry. DAIR's SFT-only system on the same base scores 46.01 public, 46.57 hidden in-domain, and 15.62 OOD, all higher. Scaling the official baseline from 2B to 9B with no training already moves public clemscore from 10.67 to 31.91. The useful content is the path-dependence: broad imitation helps when participation is missing; turn-local preference helps once failures are mechanically locatable; further correction after the policy has already changed does not. None of five continuation arms beat the 50.43 development endpoint after scaling.
A 4B probe makes the same point from the other side. Low-dose SFT on 700 off-policy successful episodes dropped clemscore from 40.75 to 24.36. Success-only imitation hurts when the parent already knows how to play.
The paper lists four. Turn-local repair was only built for the Wordle family; failures that need semantic or strategic judgment were not tested. Demonstrated OOD coverage is related-variant transfer. Repair hangs off one capped broad-SFT parent, so the study cannot tell whether turn-local DPO complements a stronger acquisition stage or just patches this weak one. The official numbers validate the endpoint, not causal phase effects: adjacent checkpoint gaps mix training, sequential dependence, repeated selection, and 67-episode noise. Static preservation is aggregate. BBH can rise while IFEval falls.
The three public plain-Wordle episodes are harsher. Conditional quality is 0 at every checkpoint that still plays: protocol errors got repaired, the guessing task did not. After s=0.85 scaling, all three episodes abort, and the bad-length guess that pass 1 targeted comes back. The submitted artifact scores almost nothing on the hidden in-domain Wordle family, which is a poor match for two Wordle-only repair stages.
Public clemscore correlates with closed in-domain at r=0.93 and with closed OOD at r=0.30. Every pipeline decision was made on the public-dev set. The endpoint looks like that signal: a large in-domain gain, and OOD Played below the untrained base.