lm-eval Pitfall: Chatty Models Break Letter-Only Prompts, Regex Grabs Wrong Answers
mancubus77 · reddit · 2026-11-05
While benchmarking LLMs on a domain-specific multiple-choice task with lm-eval, the poster hit a real issue: despite a prompt saying "respond with ONLY the answer letter," heavily instruction-tuned models often return full explanations instead. The ([A-F]) regex then grabs the first matching letter anywhere in the prose — often from an unrelated word like "designed" — so the benchmark ends up measuring which model puts a letter near the start of its response, not actual domain knowledge. The poster is asking the community for established fixes.
More from coding & agent
- Garry Tan Spends $200K/Year on Tokens With 23 Claude Code Skills — Do Agent Setups Actually Help? — gdequeiroz · 2026-11-05
- Axiom turns entire Stacks Project into 16M lines of machine-checked Lean with agents — LinusEkenstam · 2026-11-05
- OpenAI upgrades Codex: marked-up images now pinpoint edits via API — OpenAIDevs · 2026-11-05
- Baseten Router cuts coding-task costs 72% with only 0.8pt pass-rate drop vs Opus 5.5 — baseten · 2026-11-05
- BrowserForge: open-source MCP server gives agents your real browser with human approval gates — mostafazog · 2026-11-05
- Free course teaches analytics engineers AI skills beyond Snowflake and dbt — Zachly · 2026-11-05