OrcaRouter stress-tests JEV: dropping autoregressive decoding could cut inference cost 10-100x

Dan_Jeffries1 · x · 2026-09-23

OrcaRouter spent days reproducing, breaking, and improving JEV internally. Key takeaways:

The core insight is probably right: if the answer space is bounded, don't generate token by token — removing autoregressive decoding can cut inference work by 1–2 orders of magnitude.
Data, not RLCD, is the moat: Laya already open-sourced the implementation and weights; the missing piece is the synthetic data recipe, and OrcaRouter's experiments point the same way.
"Open source already beat JEV" is a benchmark illusion: the same checkpoint scores 0.769 in-distribution but only 0.541 OOD — the apparent breakthrough largely disappears under a different distribution.
Compute-optimal ≠ learnability-optimal: moving state outside the problem sequence to save FLOPs cost 27 points.

More from Models