Does grammar-constrained decoding hurt small or quantized models? A call for real numbers

bye_ella · reddit · 2026-11-16

Prompted by recurring advice that small models should just use JSON for tool calls, the author asks whether GBNF/JSON-schema constraints hurt accuracy on small quantized models (e.g. 3B at Q4), citing prior work showing strict format restrictions can degrade reasoning. No systematic with/without-constraint comparison exists yet; the author solicits real measurements, whether lower quants worsen it, and whether free-thinking-then-constraining-only-the-final-answer fixes most of it.

Original post →

More from coding & agent

coding & agent channel →