CriticNPU: Mechanistic critic halves simulator calls for sample-efficient NPU design-space exploration

Yicheng He · hf · 2026-11-19

Researchers at UIUC propose CriticNPU, a sample-efficient design-space exploration (DSE) framework for NPUs serving LLM inference.

Problem: DSE today relies on expensive simulators returning only scalar feedback (latency, power), which ranks configurations but explains neither why one is slow nor which parameter to change next—especially since prefill and decode are bottlenecked by different resources (compute throughput vs HBM bandwidth).

Approach: An online-calibrated mechanistic critic grounded in explicit hardware limits—filtering weak candidates before simulation, then aggregating operator-level runtimes into prefill/decode bottleneck summaries and predicting the effect of parameter changes. A Roofline-style model derives compute/memory limits from architecture parameters, calibrated per operator type with simulator observations, paired with a workload extractor and architecture planner.

Results: Across six LLM workloads with 16 deployment configurations each, CriticNPU achieves best-or-tied-best latency in 74% of prefill and 87.5% of decode settings while using roughly half the simulator calls of competitive DSE baselines. Code is open-sourced on GitHub.

Original post →

More from Infra

Infra channel →