Fable 5.2 aces a MazeBench level 'suspiciously' well — fueling test contamination jokes
aiamblichus · x · 2026-11-17
Blogger patiencecave found Fable 5.2 performs "suspiciously" well on MazeBench: on a 3D level it recited the solution to the old version of the level — but the level was updated weeks ago and the old solution no longer works, and the model failed to adapt. Quoters joked that memorizing benchmarks is only "benchmaxxing" in one region, "otherwise it's elegantly including the whole light cone in the training set" (meme: "I did not train on test"). Unverified but a notable contamination anecdote.
More from Fun
- ChatGPT solved a laptop-to-screen hookup that neither the author nor building management could — peterwildeford · 2026-11-17
- Dev runs Grok bots from his car via Tailscale, while rival touts $200 Claude 3-agent setup — haydendevs · 2026-11-17
- Reuters: Bessent Says NVIDIA Must 'Go Outside the Company' for New CEO — OwariDa · 2026-11-17
- Eric Jang Joins the Jensen Huang Meme Wave: Send Him a White Leather Jacket — ericjang11 · 2026-11-17
- Xue Manzi Rallying 100+ Chinese Entrepreneurs for 'Stand with Jensen' Petition — ceciletamura · 2026-11-17
- Authors say they reproduced their ICLR 2027 reviewer's erratic reviews with a simple GPT prompt — anand_bhattad · 2026-11-17