Benchmark Heaven adds benchmaxxing signal to spot models gaming famous benchmarks

airesearch12 · x · 2026-09-18

• Benchmark Heaven (benchmarkheaven.com), a model ranking aggregator, now includes a benchmaxxing signal: it measures when a model ranks higher on famous benchmarks than on held-out signals like DesignArena Elo or Epoch's software ECIs.
• The thread uses it to argue DeepSeek's new model may be somewhat benchmaxxed.
• Beyond benchmaxxing detection, the site offers filters for hosting region/company origin (China/EU/US), data confidentiality policies, multiple input/output price bases, open-weights-only views, and cost-vs-capability comparisons across providers — a practical tool for realistic model evaluation and procurement.

Related event: Benchmark Heaven Launches Open-Source Aggregator for 800 Models' Benchmarks and Real Task Costs(12 posts)→

Original post →

More from Models

• LightOnOCR-3 Reads 16th-Century Manuscripts and Ancient Greek Without Training — IgorCarron · 2026-11-02
• 'You Can Feel the Gradient Slopes': Dev Contrasts Benchmaxxed RL Models vs General Ones — theteknosaur · 2026-11-02
• LightOnOCR-3 Launches: Tops FRBench, Adds Layout Structure and Chart Extraction — IgorCarron · 2026-11-02
• NVIDIA's Kumo Is a Family of Four Tabular Foundation Models, Not Just One — pandeyparul · 2026-11-02
• SemiAnalysis: safety classifiers are underrated — Meta's Muse tops cyber benchmark at 35.2% — ThomasScialom · 2026-11-02
• DeepSeek V4.1 Flash 1017 wins over a longtime closed-source-only user: fast, cheap, reliable — _bachrc · 2026-11-02

Models channel →