Inkling 2 Small details: 594B-param MoE with 17B active, ViT encoder is back
thoma_gu · x · 2026-11-08
Technical notes on Inkling 2 Small: a 48-layer decoder-only transformer with 594B total / 17B active params, sparse MoE routing each token to 8 of 384 experts, hybrid local-global attention, up to 1M-token context, and a return to a 292M ViT encoder (20×20 patches, 2×2 pooling) after Inkling 1's encoder-free hMLP approach. Training used 12.2T tokens for pretraining plus 3T for RL. giffmana approves: "ViT is back."
Related event: Inkling 2 Small Architecture Revealed: 594B Params, 17B Active(2 posts)→
More from Models
- User accuses Anthropic of silent quantization "psyop," says it erodes trust across AI field — ryunuck · 2026-11-09
- Researcher Says Anthropic's Bio Safety Filter Blocks Codebase Reasoning, Causing ICLR Errors — josephdviviano · 2026-11-09
- Dev: with latest-gen LLMs he stops overspecifying and lets the model riff — jongranskog · 2026-11-09
- Google quietly reversed its privacy pledge: consumer Gemini now routes Gmail/Drive data to human review — tlislo · 2026-11-08
- Pro User: Gemini Broke a Month-Old Project Overnight, Burned 3 Hours of Credits Fixing It — FillerKill · 2026-11-08
- Locally-run Qwen hallucinates a realistic provider API error out of thin air — little_raven_black · 2026-11-08