Matryoshka LM Suites: Train an Entire Model Suite in One Nested Run

Researchers propose Matryoshka LM Suites: instead of training each model in a suite independently, 500M, 1.5B and 3B models are nested in a single architecture and trained in one run. Per @willdepue, the method saves roughly 36% of training compute; per @yoavartzi and @nthngdy, it benefits both training and inference, marking a notable shift in how model suites are built.

Confirmed

• Each sub-model feeds its output to the next sub-model's layer stack; unlike prior work, each sub-model can have independently tunable width and depth, opening a large design space for memory and compute.
• All sub-models train in one process, enabling free teacher-student online distillation during pretraining and higher alignment among sub-models.
• At inference, smaller sub-models serve as drafts for speculative decoding of the largest model; nested sharing of memory and activations amortizes draft overhead, yielding 10-30% faster decoding than baseline.
• Performance matches a standard suite trained on the same data and consistently beats iso-FLOPs suites (measured by total suite FLOPs); it also outperforms MatFormer on size-performance trade-offs, with greater size flexibility and variable KV Cache needs.
• Matching KV Cache footprint is found to match performance levels.

Why it matters

• One training run yields a full size-adjustable suite, cutting training compute and enabling flexible model-size selection per deployment.
• Nested sharing nearly amortizes the draft cost of speculative decoding, offering a low-cost path to faster decoding.

2026-08-19 ~ 2026-08-21 · 8 related posts

• Matryoshka Enables Free Student-Teacher Distillation During Pretraining — nthngdy · 2026-08-19
• Matryoshka Beats MatFormer in Size-Performance Tradeoff — nthngdy · 2026-08-19
• Matryoshka Matches Standard Model Performance with Flexible Width/Depth — nthngdy · 2026-08-19
• [source] Matryoshka Suite: Submodels with Adjustable Width and Depth — nthngdy · 2026-08-19
• Matryoshka: Speculative Decoding Speeds Up Inference by 10-30% — nthngdy · 2026-08-19
• Matching KV Cache Footprint Matches Performance Level — nthngdy · 2026-08-19
• [source] Matryoshka LM Suites: Train model suites in a single run — yoavartzi · 2026-08-19
• [source] Matryoshka Language Model Suites Cuts Training Compute by 36% — willdepue · 2026-08-21