Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrieval
Jianzhi Yan, Zhiming Li, Le Liu, Zike Yuan, Shiwei Chen, Youcheng Pan, Buzhou Tang, Yang Xiang, Danny Dongning Sun
ACL 2026 Findings
cs.AI
2026-04-07
DIN-Retrieval fetches ICL demos in a polarity-stable neuron subspace. Nine 7B–32B models average +1.8 over strong retrievers; LLaMA-3.1-8B gains +7.0 on GSM8K→FOLIO vs zero-shot.
Zero-shot LLMs on math and formal logic skip intermediate implications, fail to join branches, or ignore blocking conditions. A cross-domain demo with the same topology can put that skeleton back. Hand-picking by topology does not scale, and semantic retrieval follows surface wording.
The same group’s empirical study already saw conditional positive transfer: cross-domain ICL helps when the model is large enough and the demo skeleton matches. This paper, in ACL 2026 Findings, asks how to fetch those structurally close source examples automatically.
Domain-invariant neurons (DIN) are defined per layer. For each input, token hidden states at a layer are averaged into a neuron activation vector. Source and target each get a z-score per neuron against the pooled mean and variance. Dimensions that sit above τ on both sides, or below −τ on both sides, keep a stable polarity and enter the candidate set. If the set is too large, top-K is taken by |zS| + |zT|, with K a fraction of the hidden size.
A sample’s DIN vector concatenates activations on those invariant dimensions across layers. The target query is scored against the source pool by cosine similarity, then Maximal Marginal Relevance knocks down near-duplicates. k=2 source demos go into the prompt. The LLM stays frozen; only the retrieval key changes.
Why trust these neurons: on the last six layers of LLaMA-3.1-8B, pruning DINs raises perplexity more than pruning the same number of random dimensions. Relative PPL on the source rises 5.2%–8.1%. At layer −6, DIN pruning adds 7.99% PPL against 0.07% for random pruning (p=0.0332).
Four directions: FOLIO↔GSM8K and GSM8K↔PrOntoQA, on models from Qwen-2.5-7B through Gemma-3-27B and LLaMA-3.1-8B. Baselines are zero-shot, X-ICL (model-written natural-language explanations as demos), and Set-BSR (coverage of query semantics via bidirectional similarity).
| Model | Direction | Zero-shot | DIN-ICL |
| Qwen-2.5-14B | GSM8K→FOLIO | 67.4 | 70.4 (+3.0) |
| Qwen-3-8B | GSM8K→FOLIO | 81.7 | 85.8 (+4.1) |
| Gemma-3-12B | GSM8K→FOLIO | 61.0 | 65.5 (+4.5) |
| LLaMA-3.1-8B | GSM8K→FOLIO | 56.3 | 63.3 (+7.0) |
Versus zero-shot, most Qwen and Gemma models average +0.5 to +2.0. Large models already sit high; Qwen-3-32B only adds +0.7 on average. X-ICL is unstable: LLaMA-3.1-8B drops 7.8 points on GSM8K→PrOntoQA, while DIN stays near the zero-shot floor. Versus Set-BSR, DIN is 0.8–1.5 points higher on average and about 3 points on high-shift hops; LLaMA-3.1-8B averages +4.1. The abstract’s “+1.8 over SOTA” refers to this retrieval comparison.
Case studies recover GSM8K items that match a linear chain, a binary fork, or a blocking condition. Whatever link FOLIO zero-shot was missing, the demo supplies it. Using DIN-selected neurons for ICL also beats a random neuron subset of the same size (62.8 vs 59.5 on Qwen-2.5-7B, GSM8K→FOLIO).
The retrieval key moves from “do the sentences look alike” to “which neurons keep polarity across domains.” It is reproducible in the engineering sense: one pass of unlabeled source and target activations cuts out a subspace, with no weight update. The main payoff is on 7B–14B models whose zero-shot logic is shaky, especially high-shift hops such as GSM8K→FOLIO. Above 32B the baseline is already strong, and another point has to be worth the activation statistics.
This is the retrieval answer to the empirical paper, and the prompt-side foil to CoDA. DIN still stuffs cross-domain examples into the context. CoDA treats text-level gains as tapped out and edits hidden states instead. Same problem, one layer apart.
The authors are blunt. DIN identification is a fixed-threshold polarity rule and will miss richer invariances. Experiments stop at GSM8K, PrOntoQA, and FOLIO; no commonsense, no code. The causal status of the neurons is still early: lesioning hurts, which does not prove they encode a transferable topology. Gains are modest overall, k is locked at 2, and shot count is never swept.
One method detail: identifying DINs uses target-domain activation statistics, so this is not target-zero-sample in the strict sense. A new domain still needs a pool of unlabeled questions before the subspace can be cut. Same assumption as CoDA’s MMD: you already hold a target question pool.