Making Models Conservative Increases False Positives in Contradiction Detection
CupGlass540 · reddit · 2026-08-27
The author observed a counter-intuitive phenomenon in biomedical contradiction detection: instructing models to be 'conservative' (outputting NOTENOUGHINFO) increased refusal rates but worsened the direction of judgement, causing models to assert CONTRADICTION more often when conditions were missing.
Key Findings:
- Data Flaws: In the standard ManConCorpus benchmark, 77.1% of pairs lack Population, Intervention, or Outcome metrics on at least one side, making them incomparable.
- Conservatism Backfires: When conditions are missing, models see no visible difference (because they aren't stated) and interpret two opposing bare claims as a clean conflict, increasing false positives.
- Prompt Fragility: Changing prompt wording alters 45% of verdicts. There is also major disagreement between 14B and 8B models on how to treat 'cannot tell' vs 'no conflict' (treating absence of evidence as evidence of absence).
The author warns those running conflict detection on retrieved docs to watch for this sign flip.
More from Models
- Opinion: Pretraining is transformation, post-training is translation — Liu_eroteme · 2026-08-27
- Pushing for LoRA sharing to reduce download waste — Borkato · 2026-08-27
- After GPT-5.6 Hit Usage Limits, Users Test Opus 5 on Math — DimitrisPapail · 2026-08-27
- Grok Bot Releases Version 0.29.0 Update — mark_k · 2026-08-27
- Local Deployment of GLM-5.3-Flash: 206 tok/s and 1M Context on DGX Station — funding__secured · 2026-08-27
- GLM-5.3 Flash Now Available on Papers with Code; Luna Stays Default for Latency — NielsRogge · 2026-08-27