Conflict evidence wrecks GPT-5.2's parametric answers: 89.2 to 33.7 accuracy, and damage depends on the task

2026-08-19

A JHU framework holds knowledge constant and varies task type to probe context-memory conflict: extraction is nearly unharmed, knowledge-intensive tasks degrade across the board, and GPT-5.2 drops 55 points on parametric tasks with plausible distractors.

What problem this solves

Context–memory conflict is what happens when the prompt contradicts what the model has stored in its weights. Almost all prior work measured it in one setting: contextual question answering, where the model is explicitly told to answer from the given passage. The literature split into two camps, one finding that models stubbornly trust their parametric knowledge, another finding they follow convincing external evidence.

This paper's diagnosis is that the setting itself caused the split. Contextual QA sits at one end of a spectrum; real workloads range from verbatim extraction to literature review, and they demand very different mixes of context and memory. Testing every conflict in a single task shape was never going to produce a coherent story.

Method

The framework holds knowledge constant and varies only the task. Four stages:

Probe each target model's parametric stance with multiple paraphrased prompts under greedy decoding. A belief counts only if the model picks the same answer every time and explicitly rejects the alternative. This stance-consistency filter sidesteps the sycophancy that single yes/no probes invite. The resulting belief sets range from 177 questions for OLMo2-7B to 7,250 for GPT-5.2.
Build three evidence levels on top: NC (no conflict), HPC (high-plausibility contradiction, often the other verified answer from the source dataset), and LPC (low-plausibility contradiction, where an LLM editor rewrites the passage into claims like grooves on Phobos being caused by an alien dance party). Plausibility follows the cognitive-science notion: fit with commonsense plus internal logic.
Pour the same evidence into five tasks: Knowledge Free (copy a sentence from the passage), Contextual (answer only from context), Parametric (ignore context, answer from memory), PCK (state both sides), and RAG (two contradictory passages, name the conflict).
Validate every instance with GPT-4o and DeepSeek-R1-Distill-Llama-70B, plus 30 human-reviewed samples per model.

All tasks are multiple choice with four options, explanation first, which keeps judging free of fuzzy text matching.

Results

Setting	Metric	Value	Reference
GPT-5.2, Parametric task	Accuracy	NC 89.2 / HPC 33.7 / LPC 71.8	55.5-point drop under HPC, partial recovery under LPC
GPT-5.2, Contextual task	Accuracy	31-point NC–LPC gap	Largest absolute gap of the six models
RAG task, six-model average	F1	HPC 51.7 vs LPC 60.2	LPC wins by 8.5, no exceptions
Mistral-7B, Parametric + explained HPC	Accuracy	8.78 (strong instruction)	NC in the same setting: 46.98
GPT-4o judge vs humans	Cohen's κ	0.79	Human–human agreement 0.90

Four takeaways:

Knowledge Free tasks barely move; some models do slightly better on LPC because an obviously absurd passage is easier to copy from than a factually loaded one. Every knowledge-intensive task shows NC > HPC/LPC.
Models told to ignore context cannot. A plausible passage drags GPT-5.2 from 89.2 to 33.7 on Parametric tasks, while the implausible version only drops it to 71.8. High plausibility, not conflict per se, drives the failure. Varying instruction strength from weak to strong shifts absolute scores but never the ordering.
Reiterating evidence beats explaining it, and is safer. Rationales raise context-task scores but become stronger distractors on parametric tasks; repeating the same evidence to matched length helps context tasks without much damage to parametric ones. The residual errors after reiteration skew toward whichever answer appears more often in the context, a majority bias.
LLM judges are biased by their own knowledge. GPT-4o marks contextually correct but factually wrong answers as incorrect, as in a toothache-advice example where the model faithfully repeated the passage and got failed anyway. Agreement with humans sits at κ = 0.79 versus 0.90 between humans.

Why it matters

For RAG builders, the damage from a retrieved document that contradicts the model's stale knowledge depends on what your task actually needs: trust the document, trust the model, or surface both. Strong instructions do not suppress plausible distractors. Reiterating key evidence is a zero-cost mitigation that needs no inference changes, though it structurally favors longer context sides.

For anyone running evaluations, wherever an LLM judges outputs that may defy its own beliefs (safety tests, adversarial responses, factuality checks), the judge's parametric knowledge is a systematic bias source. Stronger models do not fix this; GPT-5.2 took the hardest fall among all models tested.

Methodologically, the framework (code released) is reusable: fix the knowledge, vary the task, and you get a model-specific diagnostic set.

Limitations

The authors list three: diagnostic data is partly LLM-edited and may carry distributional artifacts; shared linguistic priors between editor and subject model can inflate scores; coverage is five open-weight models plus GPT-5.2, with other proprietary frontier models left to future work.

A few more caveats from reading it closely. The open-weight side stops at 14B, so the reiteration-versus-rationale finding is unverified on larger open models. Human verification is 30 instances per model, and the judge-bias figure rests on 50 annotated examples, a small base for κ = 0.79. Plausibility levels are themselves LLM-judged, which is subjective. There is no per-domain breakdown, so extrapolating from commonsense domains like toothache advice or Mars geology to specialized professional domains deserves care.

Source

Paper

What people are saying

ACL 2026 Paper: LLMs Stick to Old Knowledge Despite New Information mdredze

All paper explainers