Fiddler's Dynamic Model Escalation: cheap judge scores everything, reasoning model sees only deferrals

hima_lakkaraju · x · 2026-10-17

Fiddler AI shipped Dynamic Model Escalation in its Centor Evaluation Runtime, targeting the classic eval dilemma: a large judge model is too slow/expensive for full coverage (so teams sample a few percent), while a small one is cheap but its scores are hard to trust.
Architecture: a decisioning model in the hot path scores every span/trace in under 100ms, emitting a verdict plus an uncertainty score; a reasoning model behind it only sees low-confidence cases the decisioning model defers, re-scoring them with full deliberation.
Key framing: the question isn't which judge to use, but when the judge should admit it doesn't know. Deferred cases become training data for the next decisioning model, so the system tunes itself on its own deferrals; reasoning spend tracks how unusual the traffic is, not its volume.
Fiddler claims up to 98% lower TCO vs external LLM evals, 100% trace coverage with no sampling, no external API calls, plus bring-your-own-judge support.

More from coding & agent