Largest open experiment finds dev tools are always mentioned but never chosen by coding agents
ycombinator · x · 2026-09-04
Armature has published the largest fully open experiment on Agent Discoverability, covering 300+ companies and 20+ sectors with every trace public. The headline finding: across tens of thousands of coding agent runs, many dev tools are "always mentioned, never chosen."
Methodology highlights:
- A panel of synthetic repositories mimicking real companies, across languages, stacks and personas
- Frozen persona-based prompts (vibe coder, junior, senior, enterprise) for comparability
- Real pinned-version agent CLIs like Claude Code and Codex, with judged results
Prior publications showed every model has priors, but prompts, memories/skills, repository context and public web data all influence tool choice. Leaderboards separate open-ended (greenfield) recommendations from deployment- and repository-constrained scenarios. Armature also offers vendors custom experiments and optimization to get featured by coding agents.
More from coding & agent
- GPT-6 Astra debuts at No.1 on Terminal-Bench, 1.9% ahead of Claude Fable 5.1 — sandersted · 2026-09-04
- Perplexity API lands in Stripe Projects: one CLI command provisions key and credits — jeff_weinstein · 2026-09-04
- GeoLibre R Package Hits CRAN: Full GIS Inside RStudio, Quarto and Shiny — giswqs · 2026-09-04
- Stripe's Link agent wallet gets official docs: one-time credentials let agents pay online — jeff_weinstein · 2026-09-04
- Reasoning effort switching without breaking cache is live in Codex and Claude — altryne · 2026-09-04
- Fighting semantic rot in agent memory: supersedes metadata plus weekly dedup sweeps — PennyLawrence946 · 2026-09-04