BullshitBench update: GPT-6-Astra beats all prior OpenAI models but still trails Anthropic
scaling01 · x · 2026-09-11
- petergostev refreshed the BullshitBench (a benchmark measuring how much models bluff/overstate) along with a redesigned data viewer.
- GPT-6-Astra performs notably better than every previous OpenAI model, yet doesn't quite reach Anthropic's top models.
- The author flags a judging dilemma: current judges are older models (Sonnet 4.6, GPT-5.2, Gemini 3.1 Pro); re-grading all responses with Astra-level judges would cost $3k, and switching judges would break continuity — some of Astra's Amber grades might flip to green or red under better judging.
More from Models
- AI Sextet offers 6 models free and unlimited for 14 days, including DeepSeek and Qwen — airesearch12 · 2026-09-11
- Astra Scores 83% on GauntletBench, First Computer-Use Agent to Beat Human Baseline — ducha_aiki · 2026-09-11
- Kimi K2.8 Preview rolls out: near-K3 coding performance, 1M context for all tiers — teortaxesTex · 2026-09-11
- Looking for a classifier of software engineering task shapes to pick models per task — StewartalsopIII · 2026-09-11
- DeepSeek V4 Pro API to continue after Sept 2026, billing unchanged — teortaxesTex · 2026-09-11
- DeepSeek V4.1 Flash Hits 98% of GPT-6 Astra's Score at 1.4% of the Cost in Third-Party Benchmark — ayushtweetshere · 2026-09-11