FP32 vs. Fidelity: Indie Inference Engine Author Questions His Speed Trade-Off
ZenZombie117 · reddit · 2026-10-23
An indie developer shares a direction dilemma with Xyntetik-Runner, a local inference engine built on two cornerstones: staying numerically close to reference implementations, and making inference reproducible and verifiable.
Where it stands (benchmarks in post): fidelity-wise, Runner beats llama.cpp on most models tested; speed-wise it still trails on both CUDA and Metal, with a correction release planned after 1.5.0. The 1.5.0 work includes tensor-core GEMM paths, tiled attention, CUDA graphs, MoE optimizations, improved quantized kernels, and reduced memory movement and kernel launches, with more planned across CUDA, Metal, and CPU.
The dilemma: the author originally assumed higher activation precision matters for tool calling, LoRA training/serving consistency, and long-running agent workflows—but with speed options nearly exhausted, he asks whether FP32 is worth keeping or whether the premise itself is wrong. Repo linked for feedback.
More from Infra
- Bloom Energy Q3 revenue hits $1.24B, up 138% YoY, raises FY26 guidance to $4.4-4.6B — Beth_Kindig · 2026-10-23
- 1M+ PHP services patched in 24h, but ~40M still exposed to PHP2Shell CVEs — FanaHOVA · 2026-10-23
- Trader dismisses AI circular financing theory: every new megawatt booked at rising spot prices — JOBhakdi · 2026-10-23
- Rumor: New Architectures Could Cut Cerebras K3 Wafer Needs From 60 to 10 — bookwormengr · 2026-10-23
- Nvidia plunges 7.21% to $212.31 in a day, still holding a $5.2T market cap — michalmalewicz · 2026-10-23
- Serverless Is a Billing Model: How to Price Stateful Compute by Usage — s4chinraja · 2026-10-23