MoE beats dense in matched OOD finetune: DeepSeek V3.1 hits 0.804 CE vs Llama 405B
kalomaze · x · 2026-11-05
kalomaze ran an out-of-distribution finetuning experiment — a highly structured spatial symbolic language task with audio conditioning — comparing a big MoE against dense backbones under matched conditions: same custom tokenizer (making cross-entropy directly comparable), same data, batch size, LR, and LoRA rank 16.
Results: DeepSeek V3.1 achieved the best Eval CE at 0.80396, beating Llama 405B (0.81760), Llama 70B (0.84229), and Llama 8B (0.85536), with all runs using roughly 228M total tokens and 91M supervised tokens. His takeaway: retraining on a larger backbone can be quite fruitful.
Related event: DeepSeek V3.1 beats Llama family in fair fine-tuning experiment(2 posts)→
More from Models
- 1,300+ graded word-association clues: Google models beat OpenAI, both beat Anthropic — Hardicecream17 · 2026-11-05
- Open-source PipesHub tops EnterpriseRAG-Bench at 93.24, beating OpenAI's 61 and Google's 41.87 — aakashgupta · 2026-11-05
- Investor mocks "SOTA for American open weight models" claim: still losing to China — nathanbenaich · 2026-11-05
- Ripple-65B Preview: a from-scratch dense 65B LLM trained for $32M by Ricursive — Terminator857 · 2026-11-05
- User baffled after an AI fails to grasp the ~90-year-old 'Who's on first' joke format — aronchick · 2026-11-05
- Cognition launches SWE-2 Flash, a 276B-A12B coding model that beats all US open-weight rivals — carlesgelada · 2026-11-05