OpenJev: open-source Jev API clone on DiffusionGemma hits ~170ms latency, 198/201 accuracy
Every-Comment5473 · reddit · 2026-09-18
TypeSafe launched Jev on Tuesday — you feed it data plus typed questions (yes/no, pick-one, 0–N scale) and it returns a probability per option, absurdly fast. One developer, tired of the waitlist, built OpenJev, an open-source server exposing Jev's API, based on Matt Mastracci's vLLM PR #57250 that replicates the trick on DiffusionGemma with a single denoising step:
- Drop-in compatible: change the base URL and TypeSafe SDKs just work; Apache-2.0 code, a Docker image (vLLM + API in one container), and a free hosted API (100M tokens/account, prompts not stored).
- Latency: TypeSafe cites 3–329s for frontier LLMs on the same task; Jev runs 70–500ms; OpenJev's hosted endpoint hits 170ms p50 (73ms on GPU).
- Quality: Mastracci's live evals put DiffusionGemma-as-Jev at 198/201 vs Jev's 191/201, and faster on a DGX Spark.
It's v0.1 riding an unmerged vLLM PR, with the author's RTX PRO 6000 serving production.
Related event: OpenJev Open-Source Clone Runs Jev-Style API on One RTX 3090(3 posts)→
More from Infra
- Google engineers: LLM benchmark harnesses silently drop requests — 200 QPS in, 38 out — AI Engineer · 2026-09-20
- The rig built to run Emacs and doomscroll X is now worth more than its owner's car — tetsuoai · 2026-09-19
- Apple M4 sustains 10 instructions per cycle, beating most rivals; M5 speedup explained — lemire · 2026-09-19
- Apple M6 bumps cores to 12 with two super cores; CPUs keep improving fast — lemire · 2026-09-19
- Apple M-series chips gained ~50% Geekbench 6 performance over three years — lemire · 2026-09-19
- Inside OpenAI's inference routing: why the proportional controller had to go — AI Engineer · 2026-09-19