AltRA-Test: ~100 samples suffice to decide if LLM judges can replace humans
Technion · hf · 2026-11-20
Technion researchers propose AltRA-Test, a statistical test for deciding when LLM judges can replace human annotators, even without shared annotators between humans and LLMs.
- How it works: it resamples mixed human-LLM annotation panels and compares inter-annotator agreement between all-human and mixed panels, then makes a replacement decision via a near-worst-gap quantile statistic.
- Results: evaluations show it typically needs only about 100 annotated items and makes better replacement decisions than baselines.
More from Research
- Team claims Codex cracked 35 open Erdős problems in 2 months, some Lean-verified — IgorCarron · 2026-11-20
- ForgeSystem paper proposes AI agents forging demand-specific software — Div_pradeep · 2026-11-20
- Accord: open-source dense metric mapping from a single camera and IMU — BE2R · 2026-11-20
- TwinD hits 95.6% AUROC on face synthesis deepfakes, beating prior detectors — Andrii Yermakov · 2026-11-20
- Watch a single artificial neuron compute: weights, bias, and one activation function — TinfoilTricorn · 2026-11-20
- Frostbite: A new CPU architecture discovered, designed and verified by AI — shioyama · 2026-11-20