Marin releases 535B-A23B pretraining checkpoints, admits benchmark contamination slipped past decontamination
dlwh · x · 2026-11-18
The Open Athena team released Marin Mochi 535B-A23B on Hugging Face: intermediate pre-training checkpoints from the Hero run, with no instruction tuning or safety post-training.
- 535B total parameters with 22.76B active non-embedding parameters per token (GrugMoe MoE); BF16 weights, 193 safetensors shards per checkpoint
- 57 checkpoints retained (step 6000–393491), all uploaded and independently verified; context grows from 4K to 16K/65K, up to 262K in the final phase
- Serving requires the Marin vLLM fork with split-expert loader; numerical parity not yet validated
- The team openly reports substantial contamination from common pretraining benchmarks that their decontamination procedure missed, limiting fair model comparisons; the investigation is documented in a blog post
Related event: Marin Releases All 57 Intermediate Checkpoints of 535B MoE Model(3 posts)→
More from Models
- Google ships Gemini 4 Hydrogen, a faster robotics model, plus Antigravity Robotics plugin — keerthanpg · 2026-11-18
- Epoch: GPT-6.2 Astra scores 174 on ECI, above the pre-Astra reasoning-era trendline — airesearch12 · 2026-11-18
- Claude delegates synthetic data generation to local Qwen3.8-27B, citing its own guardrails — maksym_andr · 2026-11-18
- GPT-6.2 stuck in compaction loop in Codex, burning 4.44M tokens in 9 minutes — Dear_Lock_5280 · 2026-11-18
- Inkling-2-Small runs on two DGX Spark units and is claimed competitive with DSv4.1 and GLM 5.3 — LiTianleli · 2026-11-18
- Users find GPT-6.2 keeps setting its generated games in the same fictional place — kieranklaassen · 2026-11-18