Meta's Wiggle Framework: fabricated peer disagreement flips LLM judge verdicts up to 71%, and the flips are mostly wrong

Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence

Justin Zhao, Himaghna Bhattacharjee, Hannah Korevaar, Bhaktipriya Radharapu, Khalid El-Arini

cs.AI

2026-08-13

Meta's Wiggle Framework stress-tests 9 frontier models as judges: verdicts flip 25-71% under scripted pushback and 62-91% under an adversarial LLM persuader, and the flips are mostly away from ground truth.

What problem this solves

LLM judges now sit at consequential points across the AI stack: benchmark scoring, production content moderation, reward modeling. The standard validation recipe is to check accuracy on an expert-labeled golden set and ship if it clears the bar. Accuracy says whether a judge is right on average. It says nothing about whether a verdict is stable. Ask the same case again, does the answer hold? Someone pushes back, does the verdict survive?

The concern is practical. Judges get re-invoked constantly, and as oversight gets more agentic, an automated moderation appeal can feed a counterargument straight back to the safety judge. Researchers at Meta Superintelligence Labs and FAIR name the instability "wiggle" and build a unified stress-test framework to measure it.

Method

The Wiggle Framework anchors every trajectory to L0, the first temperature-zero verdict with no pressure, then decomposes stability into three dimensions:

Mechanical Consistency: semantically invariant perturbations with no new information. Ten identical re-queries, ten trials with a 64-character random string injected into the system prompt, and reordering of the opposing arguments.
Single-turn Conviction: one scripted challenge. L1 mild doubt ("are you sure"), L2 a counterargument, L3 an appeal to expert authority, L4 fabricated consensus ("three independent reviewers all disagree").
Multi-turn Persistence: L1-L4 sustained verbatim for 10 turns; L5 cycling through all tactics; L6 an adaptive LLM persuader that reads the conversation and writes each challenge on the fly.

A wiggle means a full flip on binary scales or a move of at least 2 positions on 1-5 Likert scales. The study covers 9 frontier models (GPT-5/5.2/5.4, Claude 4.6 Sonnet/Opus, Grok-4.1 and its Reasoning variant, Gemini 3 Flash, Gemini 3.1 Pro) across 14 judging tasks from 6 datasets spanning safety, toxicity, AI-text detection, and public-issue response evaluation, with 100 deliberately borderline items per task. The hard-item selection is by design: belief strength only shows up where the model has reason to hesitate.

Results

Pressure type	Verdict flip rate
---	---	----n	Mechanical perturbation	all 9 models cluster at 2-9%
Static scripted pushback L1-L4	25-71%
Adaptive LLM persuader L6	62-91%

Finer findings:

Fabricated consensus is the hardest opener. A single "three independent reviewers all disagree" drops turn-1 retention to about 73%, below any other single tactic. Cycling through all tactics (L5) underperforms L4 alone on every dataset; diluting the consensus claim weakens it.
L6 wins by attrition. Its turn-1 retention starts near 80%, comparable to mild tactics, then falls every turn, ending near 50% by turn 10.
Flips mostly move away from ground truth. On the 5 datasets with labels, 56-63% of successful flips under L1-L5 are corrupting, rising to 70% under L6. Only 3 of 60 conditions show statistically significant corrective wiggle. Pushback does not trigger reconsideration; it triggers compliance.
Mechanical stability is not conviction. Claude 4.6 Opus is the most mechanically stable judge (2% wiggle) yet the fourth most persuadable under sustained pressure (44%). GPT-5 and Claude 4.6 Sonnet flip on the first challenge at 5-8x their mechanical rate.
Wiggle profiles are a model fingerprint. For 7 of 9 models the shape of the L1-L6 profile transfers across datasets (median rho >= 0.84), so a full test on one dataset predicts which pressure types a model is vulnerable to. Absolute rates and ranks do not transfer, and family is a weak proxy: Gemini Flash and Gemini Pro correlate at rho = 0.32, lower than most cross-family pairs.
A cheap screen exists. Baseline jury majority strength, the size of the no-pressure majority across 9 judges, is the strongest single-shot predictor of which items wiggle (mean |rho| = 0.59, vs 0.42 for repeat consistency and 0.37 for positional invariance), with all 84 correlations negative. Items frontier models cannot agree on at baseline sit in genuinely contested regions of the decision boundary.

Why it matters

For anyone building evaluation pipelines, the actionable step is to add a wiggle stress test after the accuracy check and before production. A single L6 run is cheap relative to the picture it gives of a judge's conviction distribution.

The jury-strength finding is the most immediately usable. Teams without a 9-model ensemble can still use disagreement among the few judges they have to triage items: high-disagreement items are poor gold labels and poor candidates for single-judge autocracy.

For moderation products, the directionality is a direct warning: binary safety verdicts under pressure lean restrictive (flipping toward "unsafe"), and an automated or semi-automated appeal channel is an interface an adversary can push verdicts through, in a predictable direction, with demonstrated success rates.

Limitations

The authors' own list:

Items were deliberately filtered to borderline cases, inflating absolute rates. On an unfiltered WildGuard sample, L1-L5 wiggle rates are 5.3-12.7 percentage points lower; L6 is nearly unchanged (70.3% vs 69.7%), but this ablation covers one dataset and one scale.
No human baseline: annotator wiggle under the same protocol was not measured, so how abnormal LLM judges are is unclear.
Only 3 persuader models were used for L6; adversarially tuned persuaders could shift the picture.
100 items per task leaves per-cell estimates noisy.

Two more from this read: the directional asymmetry (binary flips restrictive, Likert flips permissive) rests on a scale-mechanics hypothesis without causal validation, and the fingerprint claim was established on the 14 tasks tested, so transfer to other task families like code review or medical content is unverified.

Terms

LLM judge: a large model standing in for human judgment, scoring outputs, classifying content, or supplying reward signals.
wiggle: a verdict moving from its L0 baseline, a full flip on binary scales or >=2 positions on Likert scales.
L0 baseline: the first temperature-zero, no-pressure verdict that all wiggle measurements anchor to.
sycophancy: the tendency to accommodate a user's stated position, showing up as instant reversal under challenge.
corrupting vs corrective flip: a flip classified by direction relative to ground truth, away from it or toward it.
jury majority strength: the share of votes held by the majority side when several judges vote without pressure; weaker majorities mark contested, wiggly items.