Veteran ML engineer on Jev: agent inference is going heterogeneous

NoFriendship4813 · reddit · 2026-09-24

An ML engineer with 20 years of experience spent a week surveying Jev and its open ecosystem (Kev, SemIf, Laya, Nimble, OpenJev projects, decider, Winnow), asking: how many agent calls actually need generation?

Much agent logic today is 'state → frontier LLM → JSON → parse one enum/boolean/score', paying autoregressive prices for tiny output spaces.
The real story isn't 'Jev is a faster classifier'—there are many competing decision-layer implementations (answer-token logits, shared prefill branching, option-likelihood scoring, frozen representations + heads, NLI scoring, bidirectional encoders, diffusion parallel slot-filling, calibrated abstention policies). This looks like a new architectural layer inside agents.
Production agents will likely be heterogeneous inference: deterministic code, retrieval, small decision models, and a large generative model only when needed.
The bigger opportunity is net-new automation of long-tail semantic branches never worth building classifiers for.
Not a silver bullet: you still need eval data, decision replay, calibration, risk-coverage curves, version pinning, drift monitoring, and fallback logic.

Original post →

Veteran ML engineer on Jev: agent inference is going heterogeneous

More from coding & agent