Open-weights Jebadiah v2.1 scores decisions from logits: 27B hits 57.03 on Decision Index 0.3
WebDevToday · reddit · 2026-10-11
Developer released Jebadiah v2.1 (27B and 9B, Apache-2.0, Qwen-based), open models that treat decisions as closed-set scoring rather than text generation: given structured context and a typed question with fixed options, they output per-option probabilities straight from candidate-label logits — no generation, no sampling, no parsing.
Key results on the Decision Index 0.3 public suite (self-run, official scorer):
- 27B: 57.03 vs 55.11 for v2, with gains in Knowledge (+2.34), Language (+3.89), Retrieval (+2.81)
- 9B: 47.20 vs 44.09, Retrieval +5.96
- Regressions concentrated in When2Call tool-use: -6.38 (27B) and -7.44 (9B), cause unknown; no statistical significance claimed
Contamination scan against public training sources found zero hits (text-match only). Quantization agreement tests: 27B Q80 259/260, 9B MLX 4-bit 237/260, with per-question records published. The author invites critique of the eval setup, especially the tool-use regression.
More from Research
- OpenTac auto-routes tactile sensor PCBs from a 3D model in 3 seconds, covering 93% of surface — anand_bhattad · 2026-10-23
- arXiv paper gives StopGrads theoretical grounding: convergence without an underlying loss — srush_nlp · 2026-10-23
- PEARL demo: an LLM agent that checks experiment results, revises plans, and re-tests — KevinKaichuang · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23
- Math PhD student quits academia: AI is making math unrecognizable — pickover · 2026-10-23
- ICLR Submission Found Containing LLM's Own Meta-Comment Admitting Results Weren't Significant — _onionesque · 2026-10-23