llama.cpp Adds Hy3 Inference Support

pmttyji · reddit · 2026-07-14

This GitHub PR adds support for Tencent Hy3 / HYV3ForCausalLM to llama.cpp and integrates its MTP (multi-token prediction) head into the draft-mtp speculative decoding target.

Model configurations mentioned in the post include:

• 299B MoE
• 80 layers + 1 MTP layer
• Support for being used as a draft model target for speculative decoding

Essentially, this is a runtime adaptation and performance optimization aimed at large model inference.

Original post →

More from Infra

• Tabul AI launches Metal TreeSHAP to speed up Shapley values on Apple silicon — Scobleizer · 2026-07-22
• DeepSeek-V4-Flash tops out at 770 tok/s on one B300 in a vLLM batch test — Moreh · 2026-07-22
• NVIDIA starts shipping 102.4 Tbps Spectrum-6 switches for Vera Rubin AI factories — nvidia · 2026-07-22
• Apple publishes SOC 3 audit reports for Private Cloud Compute — throwfaraway4 · 2026-07-22
• Reddit GPU renters say existing platforms only give you two of three: code, recovery, fair billing — legendpizzasenpai · 2026-07-22
• The Sandboxing Manifesto: Secure Execution Environments for Agents — spirosoik · 2026-07-22

Infra channel →