Prune, train, upcycle: one model that scales with your RAM

minhnguyen · x · 2026-11-23

minhnguyen sketches an idea: iteratively prune a large model's experts into smaller variants, train on a corpus, then progressively upcycle back — yielding a single weight set that scales up/down dynamically by RAM. The key, he argues, is identifying super weights to map which weights can be cut, possibly with light finetuning/LoRA/diffing/WSP, and ideally dropping the extra small-model training step.

Original post →

More from Infra

Infra channel →