Ablating Jev-style decision models: LoRA-only hits 84.9, beating the full recipe
vanstriendaniel · x · 2026-10-24
After a month of open Jev-style decision model experiments (drawing mainly on the paper Releasing the Inner Jev by JonathanGeuter et al.), the author's key finding: you may need less training than you think. Show options as letters A/B/C, run one forward pass, and read the letter-token scores — an instruct model this way nearly matches Jev: Gemma 4 31B scores 79.6 on OpenDecisions vs Jev's 80.1.
Ablations at 31B: training only the head with frozen backbone gives 78.9 (slightly below zero-shot); LoRA-only with the head frozen at letter rows scores best at 84.9; the full LoRA+head recipe gets 84.4 — so the gain comes from adapting representations, and the letter readout is already a good head. Other findings: short runs win (400–800 steps) while longer runs mostly lose calibration; 4B models gain nothing without extra tricks.
More from Models
- 1,626-question test: free e5-large embeddings beat OpenAI's paid tier in German and Spanish — julsimon · 2026-10-24
- Muse voice assistant insists the fake "iPhone Duo" doesn't exist — jesselyu · 2026-10-24
- SWE-sweep Leaderboard: Top Models Double Bug Finds at Half the Cost, Still Under 10% — stanfordnlp · 2026-10-24
- AI Finds Software Flaws Faster Than Fix: 45,207 CVEs Logged by Late July — rohanpaul_ai · 2026-10-24
- Kimi K3.1 Pro launches on mainland China site while overseas users still capped at K3 — teortaxesTex · 2026-10-24
- 156MB Phonon-2.5 Beats Whisper large-v3 on Multilingual ASR at 1/20th the Size — julianweisser · 2026-10-24