Meta's hLLM decodes full rankings in O(1) passes with the Hungarian algorithm, 64x faster at 28ms
_reachsumit · x · 2026-09-03
Meta's hLLM (Hungarian LLM) tackles the decoding bottleneck in generative reranking: instead of autoregressively emitting N ordinals, it reads an N×K item-position score matrix off prefill hidden states with a lightweight attention head and solves optimal bipartite assignment via the Hungarian algorithm, yielding a valid permutation by construction. With LoRA fine-tuning and teacher-ranking distillation, it reaches 28ms end-to-end inference—a 64x speedup—at teacher-level ranking quality, with full ablations.
More from Research
- Linear transforms beat deep learning (ScanVI) for single-cell batch correction, preprint shows — arjunrajlab · 2026-09-03
- SimLoss Enables Single-Pass Fine-Grained Image Captioning at Multi-Stage Quality — Suryaansh Jain · 2026-09-03
- Tenstorrent and AI & Inc launch JapanFold: free inference for open-source drug discovery models — DavidBennett__ · 2026-09-03
- Until Labs Scales Cryoprotectant Search to 250,000 Molecules With AI — NirantK · 2026-09-03
- X Debate: Is Chain-of-Thought Prompting a Form of Parameter Reuse? — aryaman2020 · 2026-09-03
- Constraining agents with LL(1) grammar + structured diagnostics: what it fixes and what slips through — Upstairs-Special-925 · 2026-09-03