Constrained decoding for function calling cuts endpoint throughput by 17% to 90%

remilouf · x · 2026-11-17

remilouf observes that you can tell an inference endpoint is using open-source constrained decoding for function calling because its throughput drops 17% to 90% versus other requests. He follows up noting that better quality in their dataset does correlate with slowdown in their benchmarks—a hefty cost to pay (context points to dottxt-related work).

A practical data point for teams doing structured output / function calling: constrained decoding guarantees format correctness but can carry a very large throughput penalty.

Related event: Constrained Decoding Detection: Function Calling Throughput Drops 17%-90%(2 posts)→

Original post →

More from Infra

Infra channel →