CodeClimb: a coding agent whose reviewer keeps or reverts GraphRAG tuning experiments
blaizedsouza · x · 2026-11-15
Developer Pauliusztin shares the feedback loop behind CodeClimb, a coding agent for tuning GraphRAG pipelines: the Pi agent implements a hypothesis, code is evaluated on dev data in a sandbox, and a reviewer agent analyzes metrics, errors, and prior hypotheses to KEEP or REVERT changes and generate next-round feedback.
Key design choices:
- Multiple GraphRAG metrics (extraction F1, graph precision/recall, retrieval) are optimized jointly; since improving one can hurt another, changes must pass acceptance guardrails and show meaningful gains before being kept
- The test set is evaluated only once after optimization ends, to avoid answer leakage
- The attached chart shows 8 overnight rounds: extraction.mentions.f1 climbed from 0.60 to a kept 0.718, with several experiments reverted, and a final test score of 0.545, highlighting the dev/test generalization gap
More from coding & agent
- Database Architect: a Claude Code agent that designs schemas before you code — adnan_hashmi · 2026-11-15
- Teknium says his Hermes agents are now far more effective than Claude Code — Teknium · 2026-11-15
- Adobe hands CEO role to Anil Chakravarthy, previews local MCP for Photoshop, Illustrator, Premiere — shashib · 2026-11-15
- Running fully local AI to build a prompt-to-music-video pipeline: a deep walkthrough — julianharris · 2026-11-15
- Dev ships first Steam game after 15 years, built on custom Rust engine with Claude — schiehll · 2026-11-15
- Open-source flight search MCP tool flags every split ticket as unprotected for agents — Fast_Entrance7839 · 2026-11-15