Open Source Ternary LLM Engine Tritium: Slashes VRAM Usage and Outperforms llama.cpp
Wide_Big_6969 · reddit · 2026-07-31
A developer open-sourced Tritium (Apache 2.0), a ternary (1.58-bit) LLM engine written in Rust/CUDA. The project aims to drastically reduce VRAM usage, disk space, and increase inference speed with minimal precision loss.
Key Performance Metrics:
- BitNet 2B4T fits into 1.71 GiB, making it 7.5x smaller than fp16.
- On a single RTX 4090, it achieves decode speeds of 280-300 tok/s and prefill speeds of 12.3K tok/s.
- In bandwidth-normalized benchmarks, it outperforms llama.cpp (474 GiB/s vs 352 GiB/s effective bandwidth).
Technical Highlights:
- SALT Quantization Algorithm: A sensitivity-allocated, layered ternarization method that keeps errors close to FP16 levels even when QAT fails.
- SALT-aware Training: Quantizes directly through the engine during training, ensuring identical weight behavior between training and deployment.
The author plans to convert Qwen 3.6 27B to ternary in the upcoming Stable v1.1 release, targeting a sub-1.0% relative held-out perplexity increase.
More from Infra
- Revisiting Lossy Verification in Speculative Decoding: Mechanisms and Failure Modes — Tianyu Wang · 2026-07-31
- Energy Consumption: Single AI Prompt vs Agentic Workflow Differs by 100,000x — AndyMasley · 2026-07-31
- UBS Chart Highlights the Central Hub of the AI Compute Supply Chain — BenBajarin · 2026-07-31
- Stacking 512GB VRAM: Developer Builds Dual-Node 8x V100 Inference Cluster — UltraFOV · 2026-07-31
- Open-source Rust GGUF runtime runNburn runs 295B model on 64GB RAM, 2.8x faster decode than llama.cpp — coderyeon · 2026-07-31
- Amazon Says AI Demand Is in Early Stages, Projects Over $220B CapEx for 2026 — inductionheads · 2026-07-31