CoQuIR: Code Quality Retrieval Benchmark
青稞AI · wechat · 2026-07-15
CoQuIR is a large-scale code-quality-aware retrieval benchmark. Instead of merely finding "semantically relevant" code, it prioritizes retrieving higher-quality implementations. The author notes that existing code retrieval evaluations mostly adopt natural language retrieval criteria, focusing only on functional relevance while ignoring potential bugs, inefficient implementations, vulnerabilities, and outdated APIs in the code. This oversight can introduce technical debt and security risks into downstream generation.
The benchmark covers 11 programming languages, 42,725 queries, and 134,907 code snippets, constructing samples around four quality dimensions: correctness, efficiency, security, and maintainability. To measure "quality preference," the paper proposes two new metrics: PPA (Pairwise Preference Accuracy) and MRS (Margin-based Ranking Score). The authors evaluated 23 retrieval models and found that most models scored below random baselines on quality metrics, indicating that traditional relevance metrics fail to reflect the ability to distinguish between good and bad code.
The paper also built Repllama-3b-quality and Repllama-8b-quality through quality-aware fine-tuning. Results show that these models improved by about 20%–30% on PPA/MRS without compromising traditional nDCG@10 and MRR scores. Downstream experiments revealed that pairing with a quality-aware retriever reduced the vulnerability rate and outdated API usage in generated code, proving that "preferring high-quality code" can effectively improve the output quality of RAG for Code.
More from coding & agent
- A better path to agent autonomy is running waves, finding friction, and iterating — JnBrymn · 2026-07-22
- Coding agents are heading toward an AI-writes, AI-reviews, human-approves workflow — aftahi_ai · 2026-07-22
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- GitHub review bot hits its PR limit and forces a 39-minute cooldown — DanielLockyer · 2026-07-22
- Max reasoning effort appears to be mobile-only in Codex Remote, not desktop — GabGarrett · 2026-07-22
- A Reddit demo argues online stores should expose carts and pricing through MCP — gelembjuk · 2026-07-22