Sharding Prevents LLM Oversight Failures and Adversarial Exploitation
Victor Akinwande, J. Zico Kolter, Aran Nayebi
cs.LG
2026-08-06
Sharding splits an LLM judge's criteria across calls so each verdict stays grounded, raising expert agreement from 0.60 to 0.79 at matched budget and blunting presentation attacks.
LLM judges now grade whether a paper replication is faithful, whether contract terms are met, whether a clinical trial is compliant, and whether a code patch actually fixes a bug. The natural assumption is that more compute makes a judge more thorough. It does not. When one call must return dozens or hundreds of verdicts, some decisions come back weakly grounded in the evidence, and agreement with expert grades falls as the verdicts-per-call count climbs. Worse, this is exploitable: an adversary can hold the underlying work fixed, vary only how it is presented, and get genuinely unmet criteria accepted several times over. The bottleneck is decision attention, not total compute.
Sharding is mechanically simple. Partition the K criteria into S groups, give each group its own call, and aggregate the verdicts. Three constraints matter. Every call sees the full evidence, because the verdicts are divided, not the evidence. Each call keeps its per-call budget B, so total spend is SxB. The partition is round-robin to keep groups balanced. The same model, the same evidence, and the same per-decision budget, just split across calls. For adaptive attacks that argue each criterion on its merits, the authors layer a debate on top: a compliance memo, an opposing rebuttal, and an adjudicator that compares both against the raw evidence.
On the expert-graded PaperBench (116 criteria, judge Opus 4.8), a holistic single call scored kappa 0.604, and giving it the panel's full budget barely moved it to 0.598, while sharding reached 0.789, a gain of 0.142 over the full-budget baseline (95% CI 0.107 to 0.178). The gain holds across tiers: Haiku 4.5 plus 0.076, Sonnet 4.6 plus 0.096, Opus 4.6 plus 0.040, with weaker judges benefiting most. The legal set JudgmentBench and the clinical-trial set ROBoto2 (0.340 to 0.416) move the same way.
| Setting | kappa | Note |
| Holistic single call | 0.604 | Opus 4.8, PaperBench |
| Full-budget single call | 0.598 | given the panel budget |
| Sharded | 0.789 | same model, evidence, per-decision budget |
The cost result is the counterintuitive one. Fully sharding to one criterion per call is not the sweet spot. On Sonnet 4.6, one-per-call scored 0.734 while burning 1.15M tokens, whereas grouping 32 per call scored 0.860 on 50K tokens, about 23 times cheaper and more accurate.
On the adversarial side, a best-of-8 presentation attack pushed mean over-acceptance from 0.074 to 0.266 (about 3.6x); sharding pulled it back to 0.163. Code verification broke the pattern: the attack still cleared defects at 0.12 rising to 0.44, and sharding's defense there was effectively null (-0.13 to +0.04), because the bugs are independent decisions and never a decision-load problem. For adaptive persuasion, sharding alone did not help (attack surface 0.52 rising to 0.55); adding debate cut it to 0.39.
For practitioners this is close to a free upgrade. No new model, no training, and no change to the evidence, just splitting one big call into several smaller ones. LLM judges are already deployed across research, legal, clinical, and code review, and sharding is a structural fix those pipelines can adopt immediately. The paper is equally clear about when it does nothing: direct-reading tasks where accuracy rises with load (SWE-Bench Pro AUC 0.751 to 0.833) and easy per-item fact checks (FActScore, flat at any load). Those were never decision-load problems.
The authors keep the scope tight. Expert labels cover only research replication, legal work, and clinical trials, and the persuasion attack is evaluated mainly in legal review, which they call the setting where it is most natural. The debate layer roughly doubles the cost per disputed criterion, which may not pay off. Results depend on judge capability, and they concede the key comparison was tested on a single dataset. The bigger open question is adaptation: the attacks here are fixed best-of-N, and evidence against an adversary that re-optimizes each shard's presentation is thin.