Project APE: 82-model matrix shows verifiers miss different errors, ensembles are cost-efficient
soumitrashukla9 · x · 2026-10-15
The Project APE team argues the value of verifier diversity is under-appreciated. They mapped correlations in what different models (and model + harness combos) miss across 82 configurations.
Key takeaways:
- Different models — especially across labs — miss different errors; the correlation matrix shows strong block structure within model families and much lower correlation across them
- Ensembles of heterogeneous verifiers therefore work great and are very cost-efficient
Practical implication: for agent evals and automated review pipelines, mixing verifiers from different labs beats stacking more of the same.
More from Research
- OpenTac auto-routes tactile sensor PCBs from a 3D model in 3 seconds, covering 93% of surface — anand_bhattad · 2026-10-23
- arXiv paper gives StopGrads theoretical grounding: convergence without an underlying loss — srush_nlp · 2026-10-23
- PEARL demo: an LLM agent that checks experiment results, revises plans, and re-tests — KevinKaichuang · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23
- Math PhD student quits academia: AI is making math unrecognizable — pickover · 2026-10-23
- ICLR Submission Found Containing LLM's Own Meta-Comment Admitting Results Weren't Significant — _onionesque · 2026-10-23