Constrained decoding for function calling cuts endpoint throughput by 17% to 90%
remilouf · x · 2026-11-17
remilouf observes that you can tell an inference endpoint is using open-source constrained decoding for function calling because its throughput drops 17% to 90% versus other requests. He follows up noting that better quality in their dataset does correlate with slowdown in their benchmarks—a hefty cost to pay (context points to dottxt-related work).
A practical data point for teams doing structured output / function calling: constrained decoding guarantees format correctness but can carry a very large throughput penalty.
Related event: Constrained Decoding Detection: Function Calling Throughput Drops 17%-90%(2 posts)→
More from Infra
- tinygrad.js runs the real tinygrad in your browser on your GPU via WebGPU — CatAstro_Piyush · 2026-11-17
- Data Center Backlash Grows Loud, With Historical Echoes of 'A Nuisance Has No Rights' — justin_hart · 2026-11-17
- Constrained Decoding for Function Calling Cuts Endpoint Throughput by 17%-90% — remilouf · 2026-11-17
- Terray taps NVIDIA CUDA kernels to speed drug-design model TerraBind up to 7x — AllThingsApx · 2026-11-17
- Big Tech locks up 60% of DRAM supply it can't deploy yet — Reddit proposal for use-it-or-lose-it auctions — politefella0 · 2026-11-17
- NVIDIA expands CUDA-X science libraries: black hole analysis drops from a month to 8 minutes — AllThingsApx · 2026-11-17