OpenAI's new Decisions API fails a simple coin-flip calibration test
JnBrymn · x · 2026-10-12
Developer JnBrymn stress-tested OpenAI's new Decisions API (gpt-6-luna) with a biased-coin setup: state that a coin lands heads X% of the time, then ask for the probability the next flip is heads. Ideal output is simply X/100.
A sweep from X=0 to 100 shows severe miscalibration:
- Below 55%, predicted P(heads) is systematically inflated, scattered across 0.4-0.9 with no clean pattern
- Above 55%, outputs saturate near 1.0
- The curve is bowed well above the ideal line through the middle range
He had previously found similar odd behavior from TypeSafe's Jev on LinkedIn, and says the Decisions API is "even stranger". The reproducible sweep exposes a systematic calibration flaw in decision APIs.
More from Models
- Normies assume OpenAI's 700-point math benchmark win came from stolen mathematician chat logs — basedjensen · 2026-10-12
- Musk: FSD's neural net ate all the C++ code, now safer than most Houston drivers — oilmutt · 2026-10-12
- Naveen Rao: animals do real-time motor control on milliwatts, tokens aren't comparable to thought — NaveenGRao · 2026-10-12
- Claude refuses to train on FineWeb dataset citing legal concerns — max_paperclips · 2026-10-12
- Spotify Uses LLM Explanations for RL Credit Assignment, Beating PPO on Delayed Rewards — _reachsumit · 2026-10-12
- User Observation: Latest Models Are Obsessed With Record-Keeping, Writing Down Who Did What — repligate · 2026-10-12