NVFP4 costs only ~3.6% extra error vs BF16, and the saturate batch inference tool is open source
vanstriendaniel · x · 2026-11-06
A follow-up to the Nemotron 3.5 Transcribe benchmark: NVIDIA's model card puts NVFP4 at only 3.6% higher error than BF16 (which took 5.0 minutes in the same test) — a small accuracy cost for much faster transcription.
The author also highlights saturate, his open-source (Apache-2.0) batch inference tool on GitHub: point it at any OpenAI-compatible endpoint (your own vLLM/SGLang server, TEI, etc.), feed it dataset rows or bucket files, and get resumable, publishable parquet output — with congestion-aware concurrency tuning, no manual parameter tweaking. Use cases include OCR, embedding, classification, translation, and synthesis.
Related event: Nemotron 3.5 Transcribe Handles 11-Hour Audio in 3 Minutes for $0.30(2 posts)→
More from Infra
- Ron Baron backs Musk's space data centers, targeting 2027-2028 solar-powered orbital launches — elonmusk · 2026-11-06
- Turso launches Singapore datacenter, devs cheer the new Asia-Pacific region — glcst · 2026-11-06
- Ex-Meta employee estimates Muse costs $150-$300 per user monthly — RihardJarc · 2026-11-06
- llama.cpp PR adds ClassLRU MoE cache: +6-7% decode speed and 8% fewer uploads vs SLRU — pmttyji · 2026-11-06
- JPM's Global Cash Trading Co-Head: AI's Circular Impact on Markets Was Under-Scrutinized — ZeroStateReflex · 2026-11-06
- LLVM Dev Meeting 2026: Building durable open-source infrastructure in the agentic era — clattner_llvm · 2026-11-06