Matryoshka LM Suites: Nested Training Cuts Compute by 36%

Matryoshka (nesting doll) models are a nested multi-model design: each sub-model feeds its output to the layer stack of the next sub-model, and each sub-model can have its own width and depth, opening up a huge design space for memory and compute budgets. According to @nthngdy, the approach pays off on both the inference and training sides, and is worth watching.

Confirmed

At inference time, a smaller sub-model serves as the draft for speculative decoding of the largest model; because the nested structure shares memory and activations, the draft overhead is amortized, yielding decoding 10-30% faster than baseline
All sub-models train in the same pipeline, enabling free online teacher-student distillation during pre-training so the sub-models stay better aligned with each other
Performance matches standard suites trained on the same data and consistently beats iso-FLOPs suites (measured by total suite FLOPs)
Compared with MatFormer, the Matryoshka suite is better on the size-performance trade-off, offering greater size flexibility and variable KV Cache requirements
The research found that matching KV Cache footprint is enough to match performance levels

Why It Matters

Sub-models with tunable width and depth tie model design directly to inference compute and memory budgets, making it easy to pick sizes flexibly per deployment conditions
Nested sharing of memory and activations nearly amortizes the draft overhead of speculative decoding — a new path to low-cost decoding speedups

2026-08-19 ~ 2026-08-21 · 8 related posts

Primary sources

Matryoshka Suite: Submodels with Adjustable Width and Depth — nthngdy · 2026-08-19 21:47 (UTC+08:00)
Matryoshka Language Model Suites Cuts Training Compute by 36% — willdepue · 2026-08-21 10:54 (UTC+08:00)
Matryoshka: Speculative Decoding Speeds Up Inference by 10-30% — nthngdy · 2026-08-19 21:47 (UTC+08:00)

Matryoshka Enables Free Student-Teacher Distillation During Pretraining — nthngdy · 2026-08-19
Matryoshka Beats MatFormer in Size-Performance Tradeoff — nthngdy · 2026-08-19
Matryoshka Matches Standard Model Performance with Flexible Width/Depth — nthngdy · 2026-08-19
[source] Matryoshka Suite: Submodels with Adjustable Width and Depth — nthngdy · 2026-08-19
[source] Matryoshka: Speculative Decoding Speeds Up Inference by 10-30% — nthngdy · 2026-08-19
Matching KV Cache Footprint Matches Performance Level — nthngdy · 2026-08-19
Matryoshka LM Suites: Train model suites in a single run — yoavartzi · 2026-08-19
[source] Matryoshka Language Model Suites Cuts Training Compute by 36% — willdepue · 2026-08-21