Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learning
Le Liu, Zhiming Li, Jianzhi Yan, Zike Yuan, Shiwei Chen, Youcheng Pan, Buzhou Tang, Qingcai Chen, Yang Xiang, Danny Dongning Sun
cs.AI
2026-04-07
On 25 source–target hops, BM25, embedding, and ConE only help past a size cutoff. Gemma-3-27B gains +12.3 on ProofWriter→LogicalDeduction; 4B and 12B often drop as shots grow.
In-context learning usually assumes expert in-domain demonstrations. Those demos often do not exist in a low-resource scientific field, a new biomedical niche, or a small legal jurisdiction. The working bet here is that tasks can share a solving skeleton even when the wording does not, so source-domain examples might still help target-domain inference.
Most ICL papers pick examples inside one domain, or at most under a mild shift. This study hardens the setting: source demos and target queries come from different benchmarks, the LLM stays frozen, and only the retriever and the shot count change. The same lab’s later DIN-Retrieval and CoDA papers both cite this one as the feasibility check.
The pipeline is plain. A source pool is indexed with one measurement function. A target query uses the same function to pull top-k source items, which are concatenated into a prompt for a frozen LLM that writes a stepwise solution. Three measurements: BM25 lexical match, bge-large-en-v1.5 dense vectors, and TopK+ConE, which reranks by conditional entropy and keeps the shots that look most informative to the model.
Backbones: Qwen-2.5 (3B–32B), Qwen-3 (4B–32B), Gemma-3 (4B/12B/27B), Llama-3.1-8B. Six datasets: GSM8K, ProntoQA, LogicalDeduction, FOLIO, ProofWriter, AR-LSAT, giving 25 source–target directions. Metrics are Exact Match and the signed gap versus zero-shot. Four questions, in order: does cross-domain ICL transfer at all, how scale changes the picture, whether extra shots help monotonically, and where the points actually come from.
For the last question, DeepSeek-V3 labels repaired samples by topology: L-chains, Y-forks, chains with a blocking condition, and a residual bucket. The check is whether retrieved demos share that skeleton with the target query.
Scale splits the story. Gemma-3-27B shows negative transfer on only five source–target pairs; 4B and 12B fail on far more. Embedding retrieval on Gemma-3-27B gives +12.3 on ProofWriter→LogicalDeduction and +10.3 on ProntoQA→LogicalDeduction. Swap in a small model and BM25 often tanks: Gemma-3-4B, ProntoQA→LogicalDeduction, BM25 sits 19.7 points under zero-shot.
Shot count is tied to capacity. On 27B, positive-transfer pairs mostly rise from 1 to 4 shots. On 4B and 12B, extra examples often flatten, and they go down on structurally mismatched targets such as GSM8K and AR-LSAT. Spearman correlation between shot count and score is near zero or negative on the small models: extra demos are not absorbed, they interfere.
The fraction of zero-shot errors that get repaired depends on the retriever and the query type. Under embeddings, L-type queries repair at 52.6%–65.8%; under BM25 that range drops to 18.4%–26.3%. The residual bucket can fall to 5.3%. Retrievers also skew toward a few chain types, so Y-forks and blocking-condition demos are rarely fetched. The points do not come from semantic neighbors. They come from the small set of demos whose topology patches a broken chain.
This leaves a crack in the claim that ICL is dead without in-domain labels. Source examples can work as reasoning scaffolding, if the model is past the absorption threshold and the retriever hits the right skeleton. Below 7B, dumping cross-domain shots is a way to buy negative transfer. At 27B, embedding retrieval on logic-deduction targets really does move by double digits.
It also writes the target for follow-up work: compare structure similarity, not only semantic similarity. DIN-Retrieval uses domain-invariant neurons as the retrieval key. CoDA leaves the prompt and edits hidden states. Both lines grow out of the failure cases in this study.
The authors list four. Tasks and retrievers stay inside this reasoning suite; code, medicine, and law are untested. “Repair” is a DeepSeek-V3 heuristic label, not a formal structural metric. Decoding is fixed, so retrieval and sampling are never unpicked. The three retrievers have complementary strengths that the paper never jointly optimizes.
The main table is a color heatmap of gaps versus zero-shot, with blank or dashed cells. Absolute scores should not be compared across models without care. AR-LSAT stays low throughout; legal analytical reasoning is not rescued by cross-domain demos.