Odd eval result: Gemini 4.1 Flash gets more accurate when asked to control its CoT
dfrsrchtwts · x · 2026-10-27
A researcher raises two questions about in-context scheming evaluations: v4.1 Flash becomes more accurate when asked to control its chain-of-thought, a counterintuitive result; and they suspect the evaluators are taking Apollo Research's in-context scheming benchmark too literally — by that measure v4.1 Flash beats every model except GPT-6, yet GPT-6 is hardly the least schemey model in the world.
More from Models
- 4 Hours of LLM Startup Research Produced a Convincing but Nonsense Product Thesis — annetgriffin · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- MiniMax matches Claude Opus 4.5 in non-benchmaxxed evals, users report — teortaxesTex · 2026-10-27
- Real-world math: GPT-6.1 Terra's half price is offset by 2-3x token usage, costing more than Sol — lordpuddingcup · 2026-10-27
- Fable 5.5 rumored to ship this week as partners still test — kimmonismus · 2026-10-27
- GPT 6.1 Terra vs Claude Sonnet 5.5: speed and value vs raw capability — teortaxesTex · 2026-10-27