Prune, train, upcycle: one model that scales with your RAM
minhnguyen · x · 2026-11-23
minhnguyen sketches an idea: iteratively prune a large model's experts into smaller variants, train on a corpus, then progressively upcycle back — yielding a single weight set that scales up/down dynamically by RAM. The key, he argues, is identifying super weights to map which weights can be cut, possibly with light finetuning/LoRA/diffing/WSP, and ideally dropping the extra small-model training step.
More from Infra
- Koboldcpp ships v1.125.1 release — Fcking_Chuck · 2026-11-23
- Fireworks AI breach hits 150k Uber drivers; US tightens compute-access rules — DavidLinthicum · 2026-11-23
- Why cybercriminals skip AI attacks: too noisy, costly tokenomics, old TTPs still work — jcran · 2026-11-23
- HiSA prunes 65-71% of Q·K computations in RoPE attention, boosting decoding throughput 41% — Promptmethus · 2026-11-23
- Token consumption is shifting from dedicated GPU to usage-based inference — AccBalanced · 2026-11-23
- Microsoft publishes Azure AI architecture design guide with Foundry Agent reference architectures — adnan_hashmi · 2026-11-23