CoDA: Towards Effective Cross-domain Knowledge Transfer via CoT-guided Domain Adaptation
Jianzhi Yan, Le Liu, Buzhou Tang, Yang Xiang, Dongning Sun, Zhiming Li
cs.AI
2026-04-21
CoDA distills source CoT into a residual adapter with MMD alignment. Gemma-3-27B averages 85.0% on eight transfers, 7.8 points above zero-shot; LoRA often falls below that floor.
When a target domain has no expert chain-of-thought demos, the usual fallback is to stuff examples from some other domain into the prompt. The same group’s earlier DIN-Retrieval paper showed that structurally close cross-domain shots help a little, and not much more: surface wording is too far apart for the model to pull out a shared reasoning skeleton from text alone.
Parameter-efficient fine-tuning is less stable still. LoRA trained on the source often collapses into source-domain style on the target, scoring below zero-shot. The live question is which layer the transfer should happen in: the prompt, the weights, or the hidden states.
CoDA splits a frozen LLM at layer l. The front block reads out hidden state h; the back block generates. Source items come with a question, a CoT trace, and an answer. Target items are questions only. A residual adapter Aθ sits on h and emits z = h + Aθ(h).
Supervision is source-only. Concatenating the question with its CoT and running the front block yields a teacher state h. Mean squared error pulls the adapted source state toward h, packing the stepwise trace into the latent space instead of regenerating a long rationale. Maximum mean discrepancy (MMD), a kernel two-sample distance, then matches the adapted source and target clouds, so unlabeled target points land in a region that looks like it already carried a CoT.
At inference the LLM stays frozen. Only the adapter is applied to a new target state. Default steering strength is 1.5 and the MMD weight is 1.0. Intervention sits in the mid-to-late blocks: layer 34 of 64 on Qwen-2.5-32B, layer 34 of 62 on Gemma-3-27B. Early layers still encode surface features; last layers already tilt toward the source vocabulary. Neither end holds the reasoning manifold.
Linear steering methods such as CAA and CoT-Vectors add a single vector. On a t-SNE plot the two domains often remain two blobs after that. The nonlinear adapter is what actually merges them.
Evaluation covers eight transfers among ProofWriter, LogicalDeduction, FOLIO, GSM8K, and CommonSenseQA, on Qwen-2.5 (3B–32B) and Gemma-3 (4B–27B).
| Backbone | Zero-shot | ConE | DIN | CoDA |
| Qwen-2.5-14B | 73.7 | 76.3 | 77.5 | 78.1 |
| Qwen-2.5-32B | 76.7 | 79.5 | 79.7 | 80.7 |
| Gemma-3-12B | 73.6 | 78.4 | 80.1 | 81.3 |
| Gemma-3-27B | 77.2 | 82.4 | 83.7 | 85.0 |
On Gemma-3-27B, CoDA is 7.8 points above zero-shot. The single hop ProofWriter→LogicalDeduction moves from 77.7 to 91.6. LoRA on Qwen-2.5-14B scores 64.1, 9.6 points under zero-shot, which is source overfitting in numbers. Trainable CoT-Vectors collapse to 27.7% on Gemma-3-27B; a single added vector is brittle once the domain shifts.
Latent alignment: before adaptation, Silhouette 0.5998, k-NN mixing 0.12%, MMD 0.0492. After CoDA: 0.0321, 12.92%, 0.0354. CAA raises mixing to 9.08% but leaves Silhouette at 0.1117, so the two clouds are still partly separate at the macro scale.
Parameter count: at 12B, CoDA uses 7.4M trainable weights and hits 80.3% OOD accuracy on the efficiency plot; LoRA uses 32.7M and lands at 64.0%. At 27B the pair is 14.5M / 82.9% versus LoRA’s 56.8M / 76.0%. Dropping the MSE term on Qwen-2.5-32B cuts 80.7 to 74.8; dropping MMD cuts it to 77.1. On Gemma-3-27B, intervening at layer 34 reaches 90.0% on ProofWriter→LogicalDeduction; shallow or final layers fall off.
Retrieval-style cross-domain ICL puts examples in the prompt. CoDA twists the hidden state instead. The target only needs unlabeled questions, not target CoT. For a vertical domain with no expert annotators, that is a safer bet than source-only LoRA.
The lift over the strongest retrieval baseline is about two to three points, and about one point over DIN. The comparison that actually moves is LoRA, plus the fact that the latent clouds do merge. Treat it as a patch on retrieval, not a jump in reasoning skill.
There is no standalone Limitations section. Hard constraints still sit in the method: MMD needs a pool of unlabeled target items, so a single new question is not plug-and-play; the intervention layer is searched per architecture, from layer 15 on 3B up to layer 34 on 32B; almost every benchmark is math or formal logic, and CommonSenseQA appears only as a source into LogicalDeduction. What logical features the adapter actually edits is left to later Sparse AutoEncoder and attention-attribution work. The introduction claims a peak gain of 12.3%, while the main table’s ProofWriter→LogicalDeduction hop on Gemma-3-27B is +13.9 against zero-shot. Those two numbers were never reconciled.