Releasing the Inner Jev: A Simple Recipe for Open Decision Models
Jonathan Geuter, Andrew Cheng, David Alvarez-Melis
cs.AI
2026-10-13
Gemma 4 31B reads option probabilities from letter logits in one pass, scoring 79.6 Capability against Jev's 80.1; about five GPU-hours of LoRA raise it to 84.4.
Routing a ticket, checking a policy, or picking a tool for an agent all end the same way: choose from a fixed list and attach a probability to each option. Generating that choice token by token spends latency on words nothing downstream reads. Jev folds the job into one forward pass. The input is a typed state, a question, and the options; the output is a distribution, and no text is generated. This setup is what gets called a System One model. TypeSafe describes a specialized architecture, a parallel sampler, and a training method named RLCD, reinforcement learning built for calibration. Weights and recipe are closed.
The JevBench board already holds several open reimplementations. Some read letter logits with no training. Some average over shuffled option orders, or score each option in its own pass. Others use LoRA or full fine-tuning. The measurement that was missing is how many points those specialized pieces actually add.
Each item is a JSON chat turn: evidence, a question, and options labeled A, B, C, with thinking disabled. One forward pass yields the last-position hidden state h, after the final norm. The language-model head rows for letters A through P, at most 16 options, are softmaxed over the K options the question actually has. Gemma keeps its native logit soft-cap, 30·tanh(x/30), before that softmax. Yes/no items are two options. Score items are ordered levels. A concurrent number readout scores 80.5 Capability on the same Gemma 4 31B, against 79.6 for letters, but each digit needs its own pass. The recipe stays with letters.
Raw probabilities run hot, so one temperature is fit per question type by negative log-likelihood on 3,884 questions drawn from the training distribution. OpenDecisions is not in that fit. Gemma 4 31B uses 6.0 for choice items and 7.2 for yes/no. On the trained Qwen models a temperature below 1 sharpens the distribution and lowers the OpenDecisions score, so the temperature is clipped at 1.
The decision head grows out of the frozen letter rows: z = W0 h + A·std(h) + b, with A and b starting at zero. Mean and standard deviation of h are computed once before training and then frozen. Rank-32 LoRA, alpha 64 and no dropout, is added to every attention and MLP projection, including Qwen's linear-attention projections, and trained jointly with the head. The loss is cross-entropy, against a soft target when one exists and against the correct option otherwise. Option order is shuffled during training. The optimizer is AdamW, at 2e-5 for LoRA and 1e-4 for the head. The rate warms up over 250 steps, holds through step 300, and decays to zero at step 400. Each step is 65,536 padded tokens, about 190 questions. Four hundred steps cover about 76,000 questions, 18% of a 417,825-question pool. Gemma 4 31B takes 78 minutes on four H200s, about five GPU-hours. Merged LoRA still serves in one forward pass. Training stops at supervised cross-entropy.
Of the pool, 300,042 questions come from decider's public mixture across 79 tasks. Civil Comments, Measuring Hate Speech, STS-B, and ChaosNLI contribute human-vote soft labels. About 50,000 further items are labeled by Qwen3.5-122B-A10B and Qwen3.6-27B. Soft targets total 117,783. Those learning rates do not help Qwen3.5-4B. That run raises the LoRA rate to 1e-4, blends each target with the trained 31B's soft prediction, and shows half the questions in the evaluation format.
OpenDecisions has 2,352 items in 11 families, matched to the published mix of JevBench v1.6: 1,178 choice, 586 yes/no, 588 score, four difficulty tiers, 23 languages, 78% English. Code labels 599 items. Claude drafts 1,443, kept when Qwen3.8-27B, gpt-oss-120b, and GLM-4.7-Flash all agree, or when two agree with mean probability at least 0.5. The 310 real requests need all three. Forty-three overlaps with the training pool are removed. Capability is (I+C)/2. I is chance-corrected accuracy, with equal weight across question types and tier weights 0.1, 0.2, 0.3, 0.4 from easy to hard. C rescales calibration onto a percentage. ECE is the binned gap between confidence and hit rate; ordered score items also use the ranked probability score, RPS. Fourteen public systems rank on this set in nearly the JevBench order, Spearman 0.90. Four of them never entered construction; a line fit on the other ten predicts their board scores with an error of 3.0 points.
After temperature scaling, zero-shot Gemma 4 31B scores 79.6 Capability (I 71.9, C 87.4) and Qwen3.6-27B scores 79.0, against Jev 1.13.0 at 80.1. Unscaled, the 31B is at 72.9. It puts 99.99% of next-token mass on the legal letters, and every backbone tested averages above 99.4%. The 12B and 4B models score 75.4 and 63.6. A base Gemma 4 31B on a plain completion prompt scores 64.0; the instruction-tuned checkpoint on that same prompt scores 78.9.
Four hundred steps take the 31B to a three-seed mean of 84.4, up 4.8±2.0 from its zero-shot score and 4.3±2.5 over Jev. Calibration is 92.3, the best in the table. Eikos-27B, previously the strongest public system, scores 83.1. The gap of 1.3±2.0 includes zero. Quyet-1.0-Large has a slightly higher I, 76.7 against 76.4, and a weaker C of 86.9, for Capability 81.8. Public systems are scored on whichever is higher, their original probabilities or a recalibration on the same held-out calibration set.
| System | I | C | Capability |
| 31B zero-shot | 71.9 | 87.4 | 79.6 |
| Jeff-1.0-Large | 76.4 | 92.3 | 84.4 |
| 27B zero-shot / tuned | 67.5 / 72.6 | 90.5 / 89.9 | 79.0 / 81.2 |
| 12B zero-shot / tuned | 64.0 / 65.5 | 86.8 / 84.6 | 75.4 / 75.0 |
| 4B zero-shot / modified recipe | 46.1 / 52.8 | 81.2 / 80.4 | 63.6 / 66.6 |
| Eikos-27B | 75.7 | 90.5 | 83.1 |
| Jev 1.13.0 | 71.9 | 88.3 | 80.1 |
Under the shared recipe the 27B gains 2.2±2.2 and the 12B moves −0.4±1.8. The unmodified 4B recipe scores 63.0, below zero-shot; the three changes reach 66.6. On the smaller models Intelligence rises and Calibration falls.
On JevBench v1.7, 1,500 items drawn separately for each system, Jeff-1.0-Large scores 81.5, 4.4 above Jev at 77.1. Quyet-1.0-Large scores 81.7. A gap of 0.2 is noise, and the checkpoint places second. On the 231-item public slice the tuned 31B answers 92.1% and Jev 86.6%. Zero-shot Gemma is already at 91.8%, deck31b at 93.1%. Capability on that slice has a bootstrap standard deviation of about 3 points.
Seven decision sets kept out of training, 9,212 items, give macro accuracy of 89.4 for the tuned 27B, 89.0 for the tuned 31B, and 88.0 for Jev. On decider's 7,800 held-out items the 31B leads Jev by 2.2±0.7 accuracy points. Mean ECE on the six hard-label sets is 0.047, against 0.066 for Jev. Six knowledge benchmarks reverse the order: Jev at 92.9, the tuned 31B at 88.8, a gap of 4.0±0.8, mostly on MMLU-Pro, MMLU, and WinoGrande.
Training only the decision head scores 78.9, below the calibrated zero-shot of 79.6. Training LoRA with the head locked to the original letter rows scores 84.9 on a single run, +0.6±2.1 versus the joint-training mean of 84.4. Dropping the teacher labels scores 82.5 (−1.9±2.2). At 3,000 steps the 31B is at 83.3, and the drop is mostly calibration. The same recipe on a base model that never saw instruction tuning reaches 70.3.
The 84.9 does not clear joint training. Half a point sits inside the error bar. The split that matters is training the backbone versus training the head alone.
The system worth deploying reads a state and returns a calibrated distribution. An instruction-tuned model near 30B already supports that readout. Temperature is what lines it up with Jev: 6.0 and 7.2 move the 31B from 72.9 to 79.6.
Another five GPU-hours of supervised LoRA go past Jev, and merging the adapter leaves serving at one forward pass. The released checkpoints are Jeff-1.0-Large (seed 1 of the 31B), Jeff-1.0-Qwen-27B, Jeff-1.0-Medium, and Jeff-1.0-Qwen-4B. Capability does not rise on the 12B. The 4B needs a different rate, target, and format to gain about 3 points. Knowledge questions still favor Jev by about 4 points. On public decision sets the accuracy edge is about one point, and the tuned 27B's macro average is slightly above the 31B.
The 84.4 sits on OpenDecisions, and that set was used to choose the recipe. On the seven public sets, 31B macro accuracy moves only from 88.4 to 89.0. The lead over Eikos, 1.3±2.0, includes zero, so a new-record claim on OpenDecisions does not hold up inside the interval. The lead over Jev, 4.3±2.5, runs roughly from 1.8 to 6.8 and stays clear of zero.
The 4.4-point gap on the official board also depends on a scoring adjustment. JevBench treats a yes/no probability strictly between 0.2 and 0.8 as an abstention and scores it wrong. For the submission those probabilities were moved to 0.801 or 0.199. The argmax does not change. The score under the abstention rule does. A 0.2 gap versus Quyet, on separate draws of 1,500 items, is a second-place result.
The 1,443 drafted items pass a panel that includes Qwen3.8-27B, and the 310 real requests are kept only when all three models agree, so the retained items are ones models already converge on. Part of the teacher signal comes from Qwen3.6-27B, which is itself one of the tuned models. Temperatures are fit on the training distribution, and the paper states that they do not always transfer to OpenDecisions. With the backbone frozen, an expected-utility loss beats cross-entropy by 0.9±0.9 accuracy points. Once LoRA is added, that objective stays inside the three seeds of the main recipe.