Cortex A/B-tests coding-agent skills and rules by replaying developer corrections before keeping them
Samuele Giampieri · hf · 2026-10-16
Independent researcher Samuele Giampieri published Cortex, a tool that measurement-gates the harness (project files, skills, rules) of coding agents like Claude Code.
- Problem: the same instruction can make an agent succeed or fail across runs, so it's hard to tell if an addition to the harness actually helps.
- Method: at session end, Cortex saves fixes as replayable tasks; across sessions it spots recurring corrections and writes one new skill or rule at a time, then runs the agent with and without the new text several times, keeping it only if it helps overall and no single task gets noticeably worse.
- Existing harness text is tested the same way before deletion, and Cortex tracks whether each piece of text was ever actually read by the agent.
More from coding & agent
- Handwired compiles plain JavaScript into an exact, fully differentiable neural net — ctjlewis · 2026-10-16
- Temporal Serverless Workers on Cloud Run: queue-driven autoscaling, scale to zero — rseroter · 2026-10-16
- Meta's Superintelligence Lab Runs 200+ Direct Reports—A Lesson for Flat Multi-Agent Systems — peterjliu · 2026-10-16
- LangChain CEO: the most interesting agent harness work lives at the harness-infra boundary — hwchase17 · 2026-10-16
- GitSwarm paper: agents share branchable Git memory, solving all 30 IMOProofBench problems in one run — anirudhg9119 · 2026-10-16
- Hands-on with Gemini 4 Argon: faster than Opus 5.5 but hallucinates more and costs more — prasenx · 2026-10-16