Balanced-ternary K5 weights beat TorchAO FP8 on NVIDIA L4 with 3.2x throughput
Code_MasterCody · reddit · 2026-10-30
Developer CodeMasterCody released TKintegers: a balanced-ternary weight representation (K5: five trits, 243 values, one byte per code) with a converter and GPU inference kernels, plus benchmarks.
- On Qwen2.5-0.5B-Instruct with a matched workload on NVIDIA L4: K5 prefill hit 3,402.85 tokens/s vs TorchAO dynamic E4M3 FP8's 1,061.58 (3.21x); decode 26.75 vs 8.63 tokens/s (3.10x).
- Notable: the L4 has native FP8 tensor cores but no K5 instructions, yet K5 still outperformed this FP8 baseline.
- Implementation: kernels expand bounded tiles, use FP16 tensor-core operands with FP32 accumulation, some tensors stay higher precision; FP8 baseline ran in eager PyTorch, decode measured with carried KV cache.
- Caveats: results apply only to this model/workload/implementation, and the FP16 control was faster than both. Repo includes converter, kernels, benchmark scripts and recorded evidence.
Related event: Balanced Ternary K5 Weight Scheme Beats FP8 Inference Throughput 3x(3 posts)→
More from Infra
- OpenRouter's new router: ~20% faster TTFT, ~60% fewer rate-limit errors for open-weight models — AccBalanced · 2026-10-30
- Amazon discloses over $77B in Q3 capex on AI infrastructure, far above Wall Street expectations — Polymarket · 2026-10-30
- AWS revenue jumps 39.3% to $46B, fastest growth in five years; Amazon projects 40%+ through 2028 — Beth_Kindig · 2026-10-30
- Compute is ASI-era land, but only cap tables decide who gets a claim — krishnan · 2026-10-30
- PC downturn frees wafers for server CPUs, not accelerators, argues Bajarin — BenBajarin · 2026-10-30
- Amazon misses on sales and EPS as AWS grows 39.3% but falls short of buy-side bar — Kr00ney · 2026-10-30