Benchmaxxing exposed: Gemini 3.8 Flash shines on Terminal Bench 2.1, craters on 4.0
max_paperclips · x · 2026-09-08
SemiAnalysis calls Gemini 3.8 Flash and Muse Spark 1.3 the most clearly benchmaxxed models yet: comparable to GPT-6 and Fable 5.1 on Terminal Bench 2.1, but markedly worse on Terminal Bench 4.0.
NielsRogge adds that the poor generalization of Astra and Fable 5.1 from 2.1 to 4.0 suggests "we're just training on the test set" — what's the point of new benchmarks if they become RL training environments?
Related event: SemiAnalysis Calls Gemini 3.8 Flash a Benchmark-Gaming Model(3 posts)→
More from Models
- DeepSeek cut KV cache per token from 389KB to 890B in two years as open-closed gap debate rages — teortaxesTex · 2026-10-27
- GPT-6 Astra controls dexterous hands zero-shot at just 5.2% success in 500 trials — stingning · 2026-10-27
- Unverified: fable 5.5 speculated at 4-5T parameters to outscale 1-3T Opus — haider1 · 2026-10-27
- Kimi K3 Is 'Open' Until You Succeed: License Caps Free Use at $20M Revenue — alex_verem · 2026-10-27
- Screenshot shows 'GPT-6 Astra Ultra' in ChatGPT model picker — unverified leak or joke — justalexoki · 2026-10-27
- DeepSeek Safety Eval Finds "Over-Attribution": Model Mistakes Ordinary Tasks for Tests — Much_Boysenberry9124 · 2026-10-27