Balanced-ternary K5 inference beats TorchAO FP8 by 3.2× on NVIDIA L4
Code_MasterCody · reddit · 2026-10-30
A developer released TKintegers, a balanced-ternary weight representation, converter, and GPU inference implementation. On NVIDIA L4 with Qwen2.5-0.5B-Instruct, K5 (5 trits per weight, 243 values, 1 byte per code) beat the TorchAO dynamic E4M3 FP8 baseline:
- Prefill: 3,402.85 vs 1,061.58 tokens/s (3.21×)
- Decode: 26.75 vs 8.63 tokens/s (3.10×)
Notably, L4 has native FP8 tensor-core support but no K5 instructions. Kernels expand bounded tiles and use FP16 tensor-core operands with FP32 accumulation.
The author candidly notes this applies only to this model/workload, doesn't establish a win over every FP8 runtime, and the FP16 control was faster than both. Repo includes converter, kernels, benchmark scripts, and recorded evidence; independent replication welcome.
Related event: Balanced Ternary K5 Weight Scheme Beats FP8 Inference Throughput 3x(3 posts)→
More from Infra
- Native 3K MiniMax H3 video on a 4090: one ComfyUI node kills the VRAM crash — friedlc · 2026-10-30
- Baseten partners with Postman to serve enterprise agents for 40M developers — baseten · 2026-10-30
- AI bubble datapoint: debt financing for AI datacenters is getting harder — Terminator857 · 2026-10-30
- Ollama says enterprise teams worldwide are moving workflows to open models — ollama · 2026-10-30
- Rogers Expands Starlink Satellite Roaming to Japan, Australia and New Zealand — XFreeze · 2026-10-30
- Corey Quinn flags macOS update silently restoring the removed eMudhra root cert — kingslyj · 2026-10-30