Study of 8,351 Claude Code plugins finds 74% of docs commits are runtime instructions
rohanpaul_ai · x · 2026-09-03
An empirical study of Claude Code plugin marketplaces (arXiv) reveals a maintenance problem ordinary tooling misses: Markdown files directly control runtime behavior.
Key findings from 8,351 plugins and 77,773 commits:
- 74% of commits labeled docs actually changed instructions Claude reads at runtime; after reclassification, the true docs share fell from 10.3% to 1.7%.
- When scripts and Markdown changed together, 78% of sampled PRs were functionally coupled — new flags, renamed model names, or changed paths often required paired SKILL.md updates.
- A stale SKILL.md can instruct Claude to call interfaces that no longer exist, with no compiler to catch the mismatch.
The author recommends treating instruction files as maintained runtime interfaces, kept in sync with scripts via automated checks.
More from coding & agent
- rakyll questions whether binary identity and attestation matter in real agentic flows — rakyll · 2026-09-03
- Where Do Multi-Agent Workflows Break in Production: State, Approvals, Eval or Rollback? — Medium-Lie8127 · 2026-09-03
- Tigerless Labs' auto-gtm drafts X/Reddit posts from your PRs, never auto-posts — Aiden_Tech_Ai · 2026-09-03
- How to measure agent quality beyond task success: dev seeks real-world eval metrics — serpratik · 2026-09-03
- Giving your Grok bot an email and a credit card: webhook routines that buy things on Amazon — jeff_weinstein · 2026-09-03
- Dev builds full multi-agent stack on Nostr relay with encrypted A2A comms and unified MCP — RileyRalmuto · 2026-09-03