Swapping gold references shifts a multilingual benchmark by up to 7.55 F1, Centific finds
CentificAIResearch · hf · 2026-10-14
Centific researchers measure the overlooked half of benchmark scores: the reference.
- Setup: Using a six-language PII benchmark's retained annotation record (two independent annotator labellings, the shipped aggregate gold, and an expert reviewer gold), they hold outputs fixed and swap references.
- Results: Scores move by 4.95 and 2.00 F1 for two outputs (95% CIs [3.23, 6.27] and [0.26, 3.37]), and up to 7.55 in the worst language. Output-vs-output comparisons shift by +2.95 (CI [+1.97, +3.87]), surviving multiple-testing correction and flipping one language's margin.
- Root cause: An undocumented aggregation default kept a single annotator's label when adjudication didn't fire, making the shipped gold a partial copy of a scored output.
- Proposal: Report a reference-sensitivity band alongside every score; they show how to compute one from any retained annotation record.
More from Research
- Laude Institute launches FrontierCS 2: agents must beat deleted paper contributions blind — lateinteraction · 2026-10-23
- RFD4-Proteina's 'underrated' docs detail partial diffusion for protein redesign — AllThingsApx · 2026-10-23
- MPI-Sec PhD Thesis on Vulnerability Detection Benchmarks Includes Pre-LLM Classic "Top Score on the Wrong Exam" — mboehme_ · 2026-10-23
- LiveNerf's 29-day data finds no nerfing of Claude Opus 5.5 — TheOnlyVibemaster · 2026-10-23
- New paper makes 9 falsifiable predictions toward a major AI-enabled terror attack by 2027-2028 — StephenLCasper · 2026-10-23
- HumanoidPose recovers humanoid robots' full-body joint states from RGB video alone — cholz · 2026-10-23