Hy team pushes back on ultra-sparse MoE: sparsity recovers efficiency, not quality
douglovefishing12 · reddit · 2026-11-22
From Hy's AMA: ultra-sparse MoE like 200B-A3B is inference-unfriendly on datacenter GPUs — it consumes memory without using FLOPs. Their scaling ladders show more sparsity helps at fixed active params, but cutting active params at fixed total (156B-A8B/A5B) should hurt quality a lot. The poster notes the trade-off may flip locally, where experts live in cheap RAM and compute is the bottleneck. Fun detail: DSA tipped the GQA-vs-MLA decision toward MLA, despite spec decoding becoming compute-bound.
More from Infra
- Gemini Ultra's $100 plan includes 1000 monthly Colab credits—about 55 H100 hours — iskander · 2026-11-22
- SpaceX lenders push back on $40B of AI chips launched into space — flngr · 2026-11-22
- Homie: an offline Mac notch voice assistant running Qwen 3.5 4B locally — KungFuSaifooo · 2026-11-22
- AMD and PyTorch ship a playbook to run GPT-OSS-20B inference locally with ROCm — PyTorch · 2026-11-22
- Microsoft's full Outlook outage PIR confirms AI tooling wasn't to blame — aronchick · 2026-11-22
- User reports 400+ tok/s with Flare inference engine — capetorch · 2026-11-22