EMNLP 2026 paper: a two-stage diagnostic for LLM reasoning-graph reliability
rohanpaul_ai · x · 2026-10-22
"Diagnostic Analysis for Large Language Model Reasoning" (arXiv:2610.27245, accepted to EMNLP 2026 Main) proposes a two-stage diagnostic evaluation of whether reasoning chains/graphs are structurally reliable in open-ended multi-hop QA.
- Stage 1 decomposes the distinct capabilities of graph construction (decomposition, dependency wiring, sub-answer generation, grounding), showing reliability varies with topology, prompt design, and model family.
- Stage 2 uses uncertainty estimation as a downstream test: generated-edge experiments show graph-based computation is sensitive to upstream errors and underperforms the gold-graph setting.
- Gold-graph experiments show graph structure adds little as mere prompt context, but improves confidence-based separation of correct vs. wrong answers and enables node-level diagnostics.
The authors note that wrong dependencies alter probabilistic factorization, incorrect node answers contaminate downstream reasoning, and missing evidence makes graphs look grounded when they aren't — while most work evaluates only downstream outputs, not the construction process.
More from Research
- Stanford's Librarian Agents: RL-curated workspaces boost long-context QA by 30+ points — khademinori · 2026-10-22
- Tinker ships StableReinforce to keep off-policy RL runs from suddenly collapsing — simonguozirui · 2026-10-22
- Forward-only pretraining matches backprop: val 3.704 in 2.5 min on 8×H100, no autograd — tensorqt · 2026-10-22
- Wikipedia editors gut mech interp article, citing preprints and blog posts as unusable — Sauers_ · 2026-10-22
- Yanai Elazar surveys concept removal in activation space at Copenhagen NLP Symposium — yanaiela · 2026-10-22
- HKU's PrismaCoder: Open Multimodal Code Models Rival Proprietary Systems — hkunlp · 2026-10-22