A 10-year-old programming puzzle turned into a contamination-free Agent benchmark
nankezhishi · reddit · 2026-10-14
Arguing that famous benchmarks get contaminated over time, the author converted the Microsoft Imagine Cup Algo Game (Herbert) into a copy-paste benchmark for LLM agents: write the shortest program in a custom language to steer a robot across a 25×25 board lighting all targets, with traps that reset lit targets and recursion terminated only by non-positive numeric args.
- Scoring: points = (lit targets ratio) × (80 + 20 × (1 − bytes/limit)); judge caps at 1M robot steps, 1M VM expansions, 4096 non-tail frames, 16 KiB source.
- Uses: compare model depth of reasoning and re-test over time to detect intelligence decay.
- Full prompt included; author reports Gemini 3.8 Flash produced a 26-char solution and GPT-6.1 Sol (High) a 20-char one.
More from coding & agent
- 10 GitHub repos revealing how Cursor, Claude, Devin and other AI tools are prompted — victor_explore · 2026-10-14
- Dimillian details building Dark Veil game and custom engine with Codex — Dimillian · 2026-10-14
- SAP paper teaches LLM agents to predict their own failure (AUROC 0.85) — SAP · 2026-10-14
- Your code comments are agent bait: delete most of them, dev argues — weswinder · 2026-10-14
- CMU paper: agent logs prove 0 of 4 violations; structured records raise detection to 83% — rohanpaul_ai · 2026-10-14
- Gemini 4 Argon pattern: server-side background execution for minutes-long reasoning runs — rseroter · 2026-10-14