DIY Jev-style classifier: shuffling options lifts accuracy from 47% to 73%
WelcomeMysterious122 · reddit · 2026-09-20
A developer built their own Jev-style classifier (reading next-token probabilities instead of parsing generated text) and open-sourced it as verdict. Key findings:
- Strong position bias: small models are highly sensitive to option order; with the question deleted, they still pick option A 73% of the time
- Fix: shuffle options, ask multiple times, and average — nearly free since shuffles run in one batch, lifting multiple-choice accuracy from 47% to 73%
- Probabilities double as model-level confidence scores
- The README also documents two testing mistakes that initially flattered the results, a methodology lesson the author says matters more than the numbers
The author admits the approach mirrors how MC benchmarks have been scored for years; they just wanted to see the numbers firsthand.
More from coding & agent
- TradingAgents: an open-source multi-agent LLM trading framework in Python — mdancho84 · 2026-09-20
- This guy used an AI agent to profile every eligible bachelor in the city for two cents — gregmushen · 2026-09-20
- OpenHarness: open-source workbench for orchestrating coding agents beyond code — dee_hw · 2026-09-20
- Lessons from a cited paper-writing LangGraph agent: token blowups, fake sources, and four fixes — Altruistic-Video-849 · 2026-09-20
- Plugin4Shell zero-click RCE hits Claude Code, Codex, Copilot and Gemini CLI days before NIST IR 8587, exposing the gap in agent authorization — docybo · 2026-09-20
- px0 editor ships git status streaming via SSE, checking just 3 files instead of polling — arpit_bhayani · 2026-09-20