HINT4 targets a hardware-invariant NVFP4: 4-bit inference via int8 tensor cores on Ampere
secemp9 · x · 2026-10-14
secemp9, after 5 months in Prime Intellect's RL residency, publishes progress report 1 on HINT4, a 'hardware-invariant' NVFP4 quantization method.
- Context: NVIDIA's NVFP4 (E2M1 weights + block E4M3 scales + FP32 tensor scale, 4.5 bits/weight) runs natively only on Blackwell tensor cores. Older GPUs like Ampere fall back to NVFP4A16 (weight-only, unpacked to FP16), paying a decode cost on every multiply.
- Idea: HINT4 keeps weights in 4 bits but performs the math on 8-bit integer tensor cores — the only tensor-core instruction family supported natively from Ampere through Blackwell — enabling efficient 4-bit inference on older cards (e.g., Kimi-Linear 48B).
- Status: Whole-model evals on a 3090 are still running; final quality/speed tables, code, and a paper come with report 2.
More from Infra
- Same model, 9x cost gap: Artificial Analysis publishes cache hit rates per provider — ArtificialAnlys · 2026-10-14
- Apple's Minions turns idle Mac GPUs into a pooled cloud for ML inference — MikePFrank · 2026-10-14
- AnythingLLM ditches Ollama for llama.cpp, with up to 2x faster inference — solyarisoftware · 2026-10-14
- LlamaForge: a browser panel for upstream llama.cpp with local agents that must quote their sources — Sleepybear2611 · 2026-10-14
- CoreWeave's GPU fleet is mostly older boxes, raising questions about depreciation math — ivan_bezdomny · 2026-10-14
- NVMe reads the 7GB checkpoint in 2s, so why does ComfyUI take 30s to load? — rguerraf · 2026-10-14