Labs now train on mini-swe-agent to game evals, says Xeophon
xeophon · x · 2026-11-02
Xeophon notes that mini-swe-agent was originally built to isolate model capabilities in a fair, unseen setting. Now labs train on mini-swe tools to boost eval scores, turning it into just one (bad) harness choice among many — no compaction, no multi-agent, no real users. Effectively another case of benchmark contamination: a measurement harness is being gamed as a training target.
More from coding & agent
- From chat memory to knowledge graph: a two-layer wiki knowledge base pattern — adnan_hashmi · 2026-11-02
- Floorplan: open-source AI tool draws interactive maps of your codebase (2.1k stars) — adnan_hashmi · 2026-11-02
- GitHub Agentic Workflows hits GA: Markdown-defined automations, team velocity doubled with 91% AI commits — AIFlow_ML · 2026-11-02
- Building a knowledge graph from unstructured data using LLM extraction — adnan_hashmi · 2026-11-02
- LFM2.5-24B gets a Hermes agent LoRA: same accuracy with 20% fewer tool calls — yb2698 · 2026-11-02
- Using AI to write custom ComfyUI nodes that handle nested wildcards — Early-Ad-1140 · 2026-11-02