SchrödingerRepo: rewriting repos exposes LLM memorization on SWE-bench
SJTU · hf · 2026-09-24
SJTU researchers introduce SchrödingerRepo, which dynamically instantiates test repositories at evaluation time via four transformation levels (problem restatement, namespace remapping, intra-file reordering, functionality-preserving rewriting) to strip familiar cues from SWE-bench Verified and SWE-QA. Across popular LLMs, removing repository-side cues consistently degrades agent performance and raises interaction costs, mostly from harder exploration and localization — suggesting coding agents partly rely on memorized repo patterns rather than genuine repository reasoning.
More from coding & agent
- One 5-Minute Voice Prompt, 12 Hours of Autonomous Claude Work: The Result Went Viral — zacharynado · 2026-09-24
- Dev: shocked how willingly everyone hands everything over to Meta's Muse agent — ow · 2026-09-24
- Creator open-sources brushstroke animation workflow built on Claude Opus 5.5 — alejandroll10 · 2026-09-24
- A Ready-to-Use Prompt That Makes Your Agent Audit Its Own API Bills — gethackteam · 2026-09-24
- Months-long Claude-built Rummy 500 game opens free on web, rebuilt with Opus 5.5 — AIandDesign · 2026-09-24
- Garry Tan: agents are reshaping startup customer acquisition in both directions — garrytan · 2026-09-24