100 coding-agent runs show a shared knowledge layer lifts pass rate from 10% to 60%
BadPuzzleheaded5764 · reddit · 2026-10-10
A developer built AgentLore, a shared knowledge layer for AI coding agents that captures engineering knowledge during one attempt and retrieves it in later sessions.
Setup: the task was a text2stl CLI (text to 3D-printable STL) that must pass 13 tests covering CLI behavior, input validation, watertight mesh geometry, dimensions, and character shapes; a local quantized Qwen3.5-35B-A3B with a 40-turn budget per run. 10 independent series × 10 runs = 100 total; each series starts with an empty knowledge base, run 1 with AgentLore disabled and runs 2–10 enabled, no hand-authored seeds.
Results: run 1 passed 1/10 (10%); runs 2–10 passed 54/90 (60%); the second run alone hit 40%; series averaged 5.5/10, ranging from 9/10 to 0/10. Concept count correlated positively but non-monotonically with success.
The author acknowledges the design can't separate knowledge retrieval from repeated attempts or stochastic variation, and asks the community how to improve controls, baselines, statistics for 10 binary-outcome series, cross-task generalization, and detection of accumulated knowledge reinforcing bad approaches.
More from coding & agent
- Alma agent edits an a16z-style video in Premiere Pro fully via computer-use — itsOmSarraf_ · 2026-10-10
- Developer lets Claude work overnight via Amp Code, self-training an on-device private classifier on a Mac Mini — iannuttall · 2026-10-10
- loop-engineering hits 11.4k GitHub stars with CLI tools for orchestrating AI coding agent loops — tom_doerr · 2026-10-10
- Your App Should Fundamentally Be a Wrapper Around Agents, Not the Other Way Around — max_paperclips · 2026-10-10
- Same coding agent hits 86% vs 60% SRE diagnosis accuracy once given cluster context — tianyin_xu · 2026-10-10
- CopilotKit open-sources OpenIntelligentUI: a generative UI framework for agents (2.1k stars) — aigclink · 2026-10-10