Testing decision model Jev: add "none of these" options and never do algebra across questions
colinmcnamara · x · 2026-09-21
Colin McNamara shares two practical takeaways from testing TypeSafe's decision model Jev: always give multiple-choice questions a "none of these" option (otherwise it forces an answer even for keyboard mash), and never do algebra across questions (a question plus its strict negation summed to 1.19). He also ran the same items through an open 27B model on his own GPU: accuracy within 1.4 points, the open model better calibrated on sentiment (0.04 vs 0.09 ECE), and Jev 1.8–4.5x faster.
Related event: Jev tested: prompt tips and an open 27B model matches it(2 posts)→
More from Models
- YOCO back in spotlight: blog breaks down cross-layer KV sharing in DeepSeek-V4.1-Flash and Gemma 4 — donglixp · 2026-09-21
- Gemini 4.0 Rumored Next: Yearly Pro Updates, Monthly Flash Releases Expected — haider1 · 2026-09-21
- SemiAnalysis' Dylan Patel: Chinese labs going closed-source, 'open is dying quickly' — ns123abc · 2026-09-21
- DeepSeek Web Output Style Reportedly Lobotomized by Safe Harbor RLHF — Old_Let6328 · 2026-09-21
- JEV opens to all with $5 free credits; $0.042 input and free output pricing — op7418 · 2026-09-21
- ChatGPT 20x Max users hit tighter limits, suspect compute favors government and API customers — BopSupreme · 2026-09-21