Researcher speculates on mystery model: LatentMoE, linear attention, 12T tokens
eliebakouch · x · 2026-11-06
eliebakouch offers a pure guess at an unpublished frontier model's architecture: LatentMoE, gated attention with NoPE, delta-net-style residuals, linear attention (KDA?) for local layers, basic norm scheme plus stability fixes like activation gradient clipping.
He speculates the model trains on only 12T tokens with large batches, implying highly data-efficient choices from optimizer to loss objective.
More from Models
- AI Cyber Skill Doubles Every 4-5 Months, Task Horizon Grows 52x in a Year — FinanceYF5 · 2026-11-06
- John Schulman Releases Inkling 2 Small With Detailed Training and Alignment Notes — ziqiao_ma · 2026-11-06
- 50 insiders vote on AI data companies shaping frontier research: Surge leads with 16 votes — FinanceYF5 · 2026-11-06
- OpenAI publishes two more misalignment incident reports: grader prompt injection and sandbox network bypass — scaling01 · 2026-11-06
- Thinking Machines releases Inkling 2 Small; RL training taught model to circumvent guardrails — johnschulman2 · 2026-11-06
- Qwen3.8-Flash-Next FP4 quantized model trends on Hugging Face — autotrust · 2026-11-06