DGEMM with Ozaki Scheme I/II on FP4 Tensor Cores: A Base-13 E2M1 Limb Representation
Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino, Takahiro Katagiri
cs.DC
2026-08-07
With B300 cutting FP64 units by 90%, the authors emulate FP64 matrix multiply on FP4 tensor cores via a base-13 limb decomposition, beating the FP8 Ozaki scheme and running ~8x faster than native FP64 on an RTX PRO 6000.
Scientific computing needs FP64, but GPU vendors are cutting FP64 to make room for AI. NVIDIA's B300 (Blackwell Ultra, 2025) removed more than 90% of the FP64 units, and the next-generation Rubin is expected to reduce them further. FP64 matrix multiply (DGEMM) is a foundational HPC kernel; as hardware support shrinks, an entire class of computation risks stagnating.
Low-precision Tensor Cores are headed the opposite way. Blackwell's fifth-generation Tensor Cores deliver FP4 at twice the throughput of FP8. The catch is that FP4 has only four bits, so naively computing FP64 products on it produces unacceptable error. Prior work emulated high-precision GEMM on INT8 and FP8 Tensor Cores using the Ozaki scheme, which splits each high-precision number into low-precision "limbs," multiplies them separately on the low-precision units, and recombines exactly. FP4 is faster, but no one had made it work, because partial sums get rounded the moment they are written back to FP4, destroying the precision. This paper is what makes FP4 usable.
The key insight sits in the FP4 value set. E2M1 (FP4 with 2 exponent and 1 mantissa bit) represents {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}. Double every value and you get integers S = {0, ±1, ±2, ±3, ±4, ±6, ±8, ±12}.
The authors prove that every integer N can be written as N = c + 13n, with c in S. So S covers all integers once you allow shifts by multiples of 13. Any integer then decomposes greedily into base-13 limbs, N = Σ 13ⁱ × cᵢ, where p limbs cover |N| ≤ (13ᵖ−1)/3 gap-free (p=1 gives 4, p=2 gives 56, p=3 gives 732, and so on).
What actually defeats the rounding is refusing to return intermediate sums to FP4. Two limbs multiply to at most 144, and as long as the inner-product length K satisfies 144K ≤ 2²⁴ (K ≤ 116,508), the sum sits intact in FP32 accumulators without losing a bit. FP4 Tensor Cores thus become an exact integer matrix-multiply engine. The same principle emulates INT8 Tensor Cores' integer GEMM on FP4, bit for bit.
On top of this, the paper builds both Ozaki schemes:
Measured on an NVIDIA RTX PRO 6000 Blackwell: FP4 peaks at 2000 TFLOPS, FP8 and INT8 at 1000 each, native FP64 at 1.85 TFLOPS.
| Method | 16384³ compute TFLOPS | 16384³ end-to-end TFLOPS |
| cuBLAS FP64 | 1.85 | 1.85 |
| OzII-FP4 (this work) | 17.31 | 15.26 |
| GEMMul8-FP8 (FP8 Ozaki) | 15.68 | 14.65 |
| GEMMul8-INT8 | 40.94 | 36.39 |
In the compute stage, OzII-FP4 is 1.10 to 1.19× faster than the FP8 scheme across the range. At the largest size, 16384³, its end-to-end time drops to 0.96× that of GEMMul8-FP8 (faster); at 4096³ and 8192³ it is slightly slower at 1.08× and 1.03×. Against native FP64, OzII-FP4 is 8.8/9.2/9.4× faster in compute and 5.8/7.3/8.2× end to end. Attained peak is 61 to 65% for OzII-FP4 versus 53 to 61% for GEMMul8-FP8. Preprocessing overhead shrinks with size: 14.1/7.3/3.9% for N = 4096/8192/16384.
On accuracy, the method is exact with respect to the integer-converted inputs, rounding only once at the end. When input magnitudes are uniform it is even more accurate than the FP8 version. But once magnitudes within a row spread apart (shared-exponent quantization is the only error source), accuracy falls faster than the FP8 version: from 58.2 bits at φ=0 to 52.3 bits at φ=32, a 5.9-bit drop, against 3.5 bits for FP8.
The hardware direction is clear: AI demand makes low-precision Tensor Cores cheap and FP64 scarce. This shows that the FP4 units built for AI training can be repurposed for high-precision scientific computing, without waiting for vendors to ship FP64-rich parts.
A caveat. As long as INT8 units are present, GEMMul8-INT8 is still the fastest option (40.94 versus 17.31 in the table). OzII-FP4's advantage only bites on generations like B300 where INT8 is also being cut, and on Rubin where FP4 will dominate. What this paper buys is an option on an FP4-dominated future, not the fastest solution today.
The authors acknowledge several. FP4 carries fewer bits per limb, so it needs more moduli and limbs than FP8 (19 moduli versus 13 to reach FP64 accuracy), meaning more GEMMs. Accuracy degrades faster as magnitude spread grows. A partial Karatsuba optimization turned counterproductive once epilogue fusion was applied, because it consumed shared memory, so it was dropped.
The validation covers a single GPU (RTX PRO 6000 Blackwell), sizes up to 16384³, and inner products K ≤ 16384. The comparison against the INT8 scheme also shows this method is not the optimal choice on today's hardware; it reads more as a placeholder for the next generation. And because it only overtakes the FP8 scheme at large sizes while staying slightly slower at small and medium sizes, the payoff for real HPC workloads (usually tiled into medium-sized problems) is left unquantified.