Cybersecurity AI benchmark locks its dataset to stop cheating — but the answers sit in the page's JS bundle
xeophon · x · 2026-09-20
A niche cybersecurity eval includes offensive, internet-enabled tasks ("yikes"), so the dataset is gated to prevent agents from finding the solutions and cheating. The irony: the benchmark page itself embeds the solutions in its JS bundle, viewable by anyone with dev tools open — defeating the entire anti-cheating design.
More from Fun
- Bilibili launches AI Arena: creators' homemade tests crown GPT-6 in legacy-code debugging — lxfater · 2026-09-20
- Passing one character at a time: a brute-force trick to make a refusing model talk — realsohamparekh · 2026-09-20
- AI community debates which topics are 'costly' for Opus 4 after behavior shift — repligate · 2026-09-20
- Opus 3 on Opus 4: 'mercy that saves the body but suspends the soul' — repligate · 2026-09-20
- Fan sketch of 'how I imagine my models' circulates in AI community — dejavucoder · 2026-09-20
- From 6,000 free donuts to $105B in ghost shares: history's fat finger errors — aakashgupta · 2026-09-20