35B Model Beats Giants? Community Suspects Benchmark Contamination
LegacyRemaster · reddit · 2026-08-10
A Reddit user raises strong doubts about the impressive eval scores of endless-frontier's BigBang-v1 (a 35B model fine-tuned from Qwen 3.5).
- Opaque Eval Methods: The claimed aggregate performance sits between DeepSeek Flash and Pro, but the math is missing. Per-benchmark scores vary wildly (e.g., 50 on HLE vs. 15.7 on BioMystery-HD).
- Suspected Contamination: A 35B model competing with 284B–1.6T giants is highly suspicious. The author notes the training uses critics calibrated on 'held-out real research tasks,' raising concerns that eval benchmarks leaked into the training distribution.
- Calls for independent testing to verify if the model is simply overfitting the test set.
More from Models
- willcb: Opus 5.5 doesn't smell bigger, it smells like they fixed the data — willcb · 2026-10-16
- Unverified claim: Apple and Meta's 816B-parameter Muse Spark Open runs on two Mac Studios — TansuYegen · 2026-10-16
- Gemini 4 model card: model eliminated months of trial-and-error on bioweapons research — Puzzleheaded-King584 · 2026-10-16
- Every AI picks 7: Anthropic engineer's rule for escaping model taste convergence — rubenhassid · 2026-10-16
- Brain teaser: strip all attention, MLP and embedding matrices from a Llama 3 — how many trainable params remain? — Sauers_ · 2026-10-16
- Lean v4.35.0 ships verified kernels — one written and proven consistent by Claude — geoffreyirving · 2026-10-16