Mistral Large 4 scores 94% on 1,273 USMLE-style questions but flips 44 at temp 0
MaziyarPanahi · x · 2026-10-31
Maziyar Panahi ran 1,273 USMLE-style medical exam questions through Mistral Large 4 via API (same prompt, temperature 0) across 3 runs: 1,183 right every time, 40 partially consistent, 50 wrong every run — and 46 of those 50 repeat the same wrong answer each run. He asks doctors what evidence they'd need before letting such a model assist with patients. Follow-up: even at temperature 0, the model changed its answer on 44 questions between runs.
Related event: Mistral Large 4 Scores 94% on 1,273 Medical Exam Questions(2 posts)→
More from Models
- SenseNova-U1.5-8B hits 125 tok/s on M5 Ultra in llama.cpp and spots a $1 receipt error — Binary_orchid · 2026-10-31
- Pundit: Anthropic killing 5-hour limit plus Fable launch makes it unstoppable — iruletheworldmo · 2026-10-31
- ChatGPT Daybreak Blue sessions may train OpenAI models even with training opt-out off — littmath · 2026-10-31
- Claude refuses to draw its own logo over copyright concerns — 141_1337 · 2026-10-31
- AI cracks a 27-year-old pillar of computer science, OpenAI and Anthropic claim — Fcking_Chuck · 2026-10-31
- Perplexity early evals: Gemini 4 Carbon tops WANDR benchmark for long-form research — AravSrinivas · 2026-10-31