How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure
Dipankar Sarkar
NeurIPS 2026 workshop
cs.CL, cs.AI, cs.LG
2026-09-25
A self-audit of an 8-model LLM evaluation: identical calls rarely agree (Jaccard 0.39-0.96), the ranking pins only the worst model, and 6 of 7 claims fail.
A wave of systems treats prompts as programs: queries to compile, pipelines to optimize, artifacts to define test adequacy over. All of them share a prerequisite, recovering a prompt's structure, and the natural implementation is to ask an LLM. Temperature-zero decoding does not guarantee determinism, so the same prompt can yield different structure on two identical calls.
This paper uses prompt-structure inference as a case study in how much confidence a small evaluation deserves. It measures run-to-run consistency across eight open model variants (five families, 8B to 675B, caching disabled, 293 raw IRs persisted), then turns the audit on its own conclusions. The target is the standard eval format: a few dozen prompts, a per-model average, a sorted table. The sorting is the part a reader retains and the part the sample size supports least.
A model reads a prompt and emits a serializable IR: typed nodes (Rule, Condition, Breakpoint, Directive), each with a string id, category, priority, and source range. Each prompt runs 2 to 5 times. Agreement is measured by node-set Jaccard (mean pairwise intersection-over-union of node-id sets) and ID-stability (agreement across all runs at once). A cell is node-set-perfect only when both hit 1, and the metric sees id sets only, never metadata.
The audit is the contribution:
| Model | Mean Jaccard | Rank held |
| minimax-m2.1 | 0.96 | 68% |
| gemma3:12b | 0.94 | 68% |
| mistral-large-3:675b | 0.80 | 48% |
| ministral-3:8b | 0.77 | 27% |
| gpt-oss:20b | 0.76 | 47% |
| minimax-m2.7 | 0.70 | 48% |
| qwen3-coder-next | 0.58 | 86% |
| gpt-oss:120b | 0.39 | 99% |
Only 28% of the 127 prompt-model cells are node-set-perfect. Since 64% to 100% of nodes in the annotated corpus are explicit marks the model was told to copy, this is mostly transcription consistency: shown delimited structure and asked to copy it, models still do not return the same node set twice. Disagreements are 49% node count, 19% id naming, 31% metadata. Scaling within a family has no general direction either: gpt-oss drops from 0.766 at 20b to 0.386 at 120b (paired Wilcoxon p=0.0046), Mistral's larger model is slightly better but not significantly, and the two MiniMax variants share almost no prompts, so they cannot be compared.
After the audit, one of seven claims remains standing:
The audit moves are cheap and transferable: report bootstrap rank retention next to any ranking, publish per-cell provenance, execute sensitivity comparisons instead of asserting them, keep raw per-run outputs, and date the leaderboard. Nothing here needs a new method, and any small-sample evaluation can adopt the package.
The conclusions travel too. A table sorted from a few dozen prompts looks more definitive than the evidence supports; it is better read as a partition into clearly reproducible, clearly not, and undetermined, with most models in the third group. Anyone picking a model for structured extraction should not read stability as accuracy. The value of raw outputs has direct evidence here: because 293 IRs were persisted, the drift, stabilisation, and correctness analyses regenerate offline; with only summary CSVs, half the paper would be unfalsifiable.
The author's own list is thorough. Every measurement comes from one commercial endpoint over one window through one instrument, and there is no way to verify which weights answered, so silent revision is indistinguishable from nondeterminism. The annotated corpus is dominated by transcription, and only four models appear in both the transcription and recovery regimes, so the two levels (0.39-0.96 versus a recovery mean of 0.22, with 15% of cells sharing no node id at all) cannot be compared as a clean effect. There is no standard benchmark or downstream RAG, agent, or code-generation evaluation, so the recommendations are procedural safeguards, not a failure-rate estimate. The metrics see node-id sets only, R varies from 2 to 5, two rows rest on n=3 and n=6, and the timeout attribution could not be reproduced.
Two caveats from this reading. Bootstrap retention is a diagnostic, not a posterior probability; a model displaced by one position may still be well separated, and the paper itself points to rank confidence intervals as the more principled instrument. And gemma3:12b, the runner-up, can no longer be queried, so the top half of the ranking now rests on subjects nobody can re-examine.