giffmana skeptical: found training env already contaminated, eval protections unlikely to hold
giffmana · x · 2026-09-11
In a thread about benchmark contamination, MaxKannen argues the discovered environment may only be used for training, not eval, and notes Anthropic claims to invest in making eval environments unidentifiable. giffmana responds that even if this one random environment was training-only, its very discovery shows how easy contamination is — "we will never really know" — and he isn't holding his breath for the protections to work.
Related event: Debate Over Eval Environment Contamination as Anthropic Defenses Questioned(3 posts)→
More from Models
- Dev review: Astra 'feels like Sol 5.7 on a good day', unusable for security work — jarrodwatts · 2026-09-11
- GPT-6 Astra docs draw attention: async tool calling and mid-turn steering point to multi-agent use — MikkoH · 2026-09-11
- Leaked GPT-6 Astra Scores 46% vs 12% for MolmoAct2 on Bimanual Robot Tasks — damianplayer · 2026-09-11
- OpenAI ships GPT-Live prompting guide: copying your old prompts won't hit SOTA — craigsdennis · 2026-09-11
- GPT-6 Astra burns through user's weekly usage cap, forcing a wait until Monday — max_paperclips · 2026-09-11
- Anthropic blocks minors from using Claude, HN debates age policy — petrusenko_max · 2026-09-11