Run a 744B MoE model on consumer hardware
Prompt Engineering · youtube · 2026-07-20
A demo of Colibri running GLM 5.2 on consumer hardware with about 25 GB of RAM.
The post highlights two key facts:
- the model is a 744B MoE
- only about 40B parameters are active per token
Because MoE routes only a subset of experts each step, the author argues that only roughly 11 GB of weights change token to token. The takeaway is that extremely large models can sometimes be made usable on modest hardware through routing and careful local execution.
Related event: Open-Source Colibri Runs 744B GLM-5.2 on Consumer Laptops(2 posts)→
More from Infra
- Inference providers built the market squeezing their margins; software is the only way out — AccBalanced · 2026-10-27
- EMNLP Oral: training-free Nexus Sampling cuts KV-cache memory 10x; OpenResearcher tops BrowseComp-Plus at 54.8% — yuz9yuz · 2026-10-27
- Firms offer neoclouds restrictive short-term GPU loans, eyeing cheap GPU pickups — AccBalanced · 2026-10-27
- Anthropic's S-1 exposes the compute financing gap: multi-year commitments vs shifting demand — AccBalanced · 2026-10-27
- Meta unveils 102.4T liquid-cooled network rack and 2-tier Flattened Planar Fabric for AI at OCP 2026 — Meta_Engineers · 2026-10-27
- huggingface_hub 2.3.0 cuts 1M-file listings to 20s, adds live Bucket following — lhoestq · 2026-10-27