Artificial Analysis Ships Intelligence Index v4.2 with Private Test Sets to Curb Benchmaxxing

On September 5, Artificial Analysis released Intelligence Index v4.2, an interim update that ships parts of the planned v5 early to keep pace with frontier model releases, eight months after v4. Tasks are harder and more realistic, 40% of the index weight now comes from private test sets to prevent gaming, and the saturated GPQA was dropped. Two new benchmarks debuted: AA-Briefcase for agentic knowledge work and GDP.pdf for long-document reasoning, alongside a full published methodology.

Confirmed

• Index changes: more complex, realistic tasks; 40% private test-set weight; GPQA removed after saturation. alexandrwang noted Muse spark 1.3 max remains strong, with the efficiency frontier largely unchanged.
• AA-Briefcase: Anthropic's Claude Fable 5.1 and Opus 5 lead the agentic knowledge-work benchmark; OpenAI's GPT-6 Astra trails by roughly 85 Elo but is catching up fast.
• GDP.pdf: cross-4,592-page long-document reasoning benchmark. GPT-6 Astra ranks first at 33.2%, GPT-5.6 Sol second at 28.2%, Claude Fable 5.1 third at 26.2%.
• Token efficiency: GPT-6 Astra leads on the token-efficiency dimension in v4.2.
• Competitive landscape: Hesamation noted the 300-person Moonshot team's Kimi is closing in on Google on the overall index.
• Methodology: a full methodology page explains how the index combines reasoning, knowledge, math, coding and other benchmarks, while acknowledging the limits of any single metric.

Why it matters

• Moving 40% of weight to private test sets and dropping saturated GPQA directly addresses benchmark contamination and benchmaxxing, and may reshape labs' optimization strategies. cHHillee cautioned that evaluators themselves must avoid benchmaxxing tendencies, not just labs.
• The two new benchmarks show a split landscape — OpenAI leads long-document reasoning while Anthropic leads agentic knowledge work — giving finer-grained guidance for model selection.
• Moonshot's 300-person team nearly matching Google on the overall index shows smaller teams can still break through.

2026-09-05 ~ 2026-09-07 · 14 related posts

• [source] Artificial Analysis Launches Intelligence Index v4.2 With 40% Private Test Sets to Block Benchmark Gaming — ArtificialAnlys · 2026-09-05
• Artificial Analysis launches Intelligence Index v4.2; GPT-6 Astra most token-efficient — ArtificialAnlys · 2026-09-05
• [source] Claude Fable 5.1 and Opus 5 Top AA-Briefcase Agentic Eval; GPT-6 Astra Gains ~85 Elo — ArtificialAnlys · 2026-09-05
• GPT-6 Astra Tops GDP.pdf Long-Document Reasoning Benchmark at 33.2% All-Pass Rate — ArtificialAnlys · 2026-09-05
• [source] Artificial Analysis discloses full benchmarking methodology spanning a dozen evals — ArtificialAnlys · 2026-09-05
• Artificial Analysis releases Intelligence Index v4.2, retiring saturated GPQA Diamond — gaotianyu1350 · 2026-09-05
• Artificial Analysis ships Intelligence Index v4.2 with private test sets to curb benchmark gaming — shuchaobi · 2026-09-05
• AA Intelligence Index v4.2 lands as 300-person Moonshot's Kimi closes in on Google — Hesamation · 2026-09-05
• Artificial Analysis Ships Intelligence Index v4.2 with Private Test Sets to Prevent Benchmark Gaming — poigre · 2026-09-05
• Artificial Analysis Intelligence Index v4.2 drops saturated GPQA, adds private test sets to curb gaming — SonglinYang4 · 2026-09-06
• Artificial Analysis rolls out Intelligence Index v4.2 with long-horizon agentic benchmarks — ArtificialAnlys · 2026-09-06
• Artificial Analysis Rushes Out Index v4.2 With Private Test Sets as Benchmarks Chase Frontier Models — solyarisoftware · 2026-09-06

2 near-duplicate retellings: ArtificialAnlys · burny_tech