Fable 5.2 aces a MazeBench level 'suspiciously' well — fueling test contamination jokes

aiamblichus · x · 2026-11-17

Blogger patiencecave found Fable 5.2 performs "suspiciously" well on MazeBench: on a 3D level it recited the solution to the old version of the level — but the level was updated weeks ago and the old solution no longer works, and the model failed to adapt. Quoters joked that memorizing benchmarks is only "benchmaxxing" in one region, "otherwise it's elegantly including the whole light cone in the training set" (meme: "I did not train on test"). Unverified but a notable contamination anecdote.

Related event: Fable 5.2's Suspiciously Good Puzzle Solving Sparks Benchmark Contamination Concerns(3 posts)→

Original post →

More from Fun

Fun channel →