HARDEN makes benchmarks harder without changing answers to test real robustness
burkov · x · 2026-11-08
Andriy Burkov highlights HARDEN, a method addressing the gap between curated benchmark examples and the ambiguity of real financial, biomedical, and legal work.
- HARDEN repeatedly generates and tests realistic variations of existing benchmark inputs while keeping expected answers fixed
- It searches for changes that reduce a model's accuracy and confidence, then filters candidates for evidence preservation, domain realism, and technical validity
- The key safeguard: avoids mistaking broken or misleading test cases for genuine robustness failures
- Tested on 200 cases
More from Research
- In vivo CRISPR base editing of ANGPTL3 shows durable lipid lowering, heading to trials — EricTopol · 2026-11-08
- Testing ICE Constrained Decoding: Enforce Constraints at Generation, Not After — manvigupta03 · 2026-11-08
- Near-identical neural net weights still diverge on ~3% of inputs after training — bravo_abad · 2026-11-08
- Removing negative-advantage samples drops robot zip-tie success from 86% to 60% — micoolcho · 2026-11-08
- DECIDE benchmark tests voice-agent decisions on 4,867 questions from 1,081 real calls, scoring 20 models — Aiden_Tech_Ai · 2026-11-08
- KV Cache Quantization Tested Across 35 Quants: 8-bit Is Essentially Free for Qwen3.8 27B — Hefty_Wolverine_553 · 2026-11-08