AngelSlim: A more accessible, comprehensive, and efficient toolkit for large model compression
Rui Cen, QiangQiang Hu, Hong Huang, Hong Liu, Song Liu, Xin Luo, Lin Niu, Yifan Tan, Decheng Wu, Linchuan Xie, Rubing Yang, Guanghua Yu, Jianchen Zhu
Hunyuan AI Infra Team
cs.LG, cs.AI
2026-02-07
AngelSlim unifies QAT, PTQ, Eagle3, sparse attention and pruning. HY-1.8B-2Bit scores 63.18 vs 67.15 FP16 and 63.31 INT4; Eagle3 speeds Qwen3 about 1.7-1.9x.
Quantization, speculative decoding, sparse attention, and token pruning each have their own papers. Putting them on one serving stack is where they step on each other: PTQ scales fail to match a draft model, sparse kernels lock to one architecture, multimodal pruners want to rewrite the Transformer loop. AngelSlim is Hunyuan's attempt to run that stack as a single pipeline, from algorithm to vLLM and SGLang.
The target is the inference wall: too many parameters, too long a context, too many vision and audio tokens for commodity GPUs and phones.
Four tracks share one toolkit.
Quantization is the heavy track. 2-bit uses QAT with SEQ, mapping weights to the symmetric set {-1.5, -0.5, 0.5, 1.5}, starting from instruction-tuned weights and recovering on about 89B high-signal tokens rather than retraining from a raw checkpoint. Ternary comes in two flavors. Tequila treats deadzone weights as a temporary bias and folds them back after training. Sherry forces exactly three nonzeros in every four weights and packs the block into 5 bits, which is 1.25-bit and SIMD-aligned. PTQ covers FP8, INT8, and INT4, including AWQ and GPTQ, plus LeptoQuant for peaky distributions: it searches a scale that isolates outliers so the mass sits in a denser region of FP8-E4M3. Calibration can offload intermediate layers to CPU, enough to quantize DeepSeek-R1 on one GPU.
Speculative decoding trains Eagle3 draft models to match the target distribution, not to write well on their own. The trainer extracts hidden states, unifies text, vision, and speech preprocessing, and exports into vLLM and SGLang.
Long-context prefill gets a training-free sparse attention library: static A-shape, Tri-shape, and Dilated masks; dynamic MInference, XAttention, FlexPrefill; and Stem, which gives early tokens a larger keep budget and reweights attention by Value magnitude so high-score, low-contribution tokens do not hog the budget.
Multimodal pruning is a function that returns a boolean mask. IDPruner iterates with MMR between saliency and diversity and does not need a full-network attention map. Samp, for audio, merges adjacent similar spans then diversity-prunes with attention weights. It sits in front of the LLM so FlashAttention's missing internal scores are not a blocker.
HY-1.8B-2Bit, QAT from the 1.8B instruct model, averages 63.18 across eight suites. FP16 is 67.15 (minus 3.97). INT4 GPTQ is 63.31 (minus 0.13). Dense HY-0.5B, similar footprint, sits at 45.28. On Apple M4, TTFT versus BF16 is about 3x to 8x for 256-1024 token prompts. On Dimensity 9500 versus 4-bit, prefill is about 2x and decode about 1.5x.
Sherry on LLaMA-3.2-1B matches Tequila at 0.519 average on five zero-shot tasks, at 1.25 bits versus 1.67. On an Intel i7-14700HX, a 0.7B model runs 148.27 tok/s at 205.5MB, against 132.13 tok/s and 256.6MB for 2-bit.
| Setup | Metric | Result | Baseline |
| HY-1.8B-2Bit | 8-task avg | 63.18 | FP16 67.15 / INT4 63.31 |
| DeepSeek-R1 W4A8-FP8 | AIME 2024 | 88.67 | FP8-Block 88.67 |
| Qwen3-8B Eagle3 | tok/s | 257.52 | vanilla 151.81 |
| Stem / Qwen3-8B | LongBench avg | 31.64 | dense 32.01 / FlexPrefill 28.55 |
| IDPruner, 10% keep | AI2D / ChartQA | 75.16 / 62.48 | unpruned 82.48 / 83.68 |
LeptoQuant lifts Hunyuan-4B-Instruct AIME 2024 from 66.70 (FP8) to 76.66 (BF16 78.30). On Hunyuan-2B AIME 2025 it drops 37.00 to 36.00, so the search does not always help. Eagle3 takes Qwen3-32B from 43.32 to 74.10 tok/s with mean accepted length about 1.91. Every speculative table is batch size 1.
This is an engineering report, not a new theory of compression. The 2-bit numbers say compressing a larger instruct model beats training a 0.5B dense model if you want reasoning on device. Treating speculative decoding as a trainable, deployable feature, including VL drafts in vLLM, is more useful than another acceptance-rate ablation. Sparse kernels and pruners decouple from the backbone, so swapping an algorithm is mostly config.
PTQ checkpoints and Eagle3 drafts are already on Hugging Face. 1.25-bit and 2-bit still need QAT.
There is almost no Limitations section. The wrap-up claims 4x edge speedup; the body shows 1.5-2x for 2-bit versus 4-bit, and 3-8x TTFT versus BF16. Those are different comparisons. Sherry's accuracy is on LLaMA-3.2 1B/3B, not Hunyuan-1.8B. Eagle3 is single-request; high-QPS serving will shrink the draft gain. IDPruner at 90% compression still drops ChartQA from 83.68 to 62.48. The 440MB 1.25-bit translator lives in the sister Hy-MT2 report.