Indie RAG eval auditor lands zero paid gigs despite reproducible findings
Additional-Ladder-24 · reddit · 2026-10-23
An independent specialist shares two public beta RAG evaluation audits: on DPOLens (30 queries), independently reproducing a benchmark showed adding a reranker dropped Recall@5 from 76.7% to 46.7%, with query-level breakdown; on RouteMind (700 queries), comparing two retrieval configs across 1,400 results surfaced 15 scoring inconsistencies the maintainer then fixed. Yet a month of outreach has yielded no paid engagement. They ask agency owners and CTOs what's missing: references, a live walkthrough, a paid trial, or whether agencies simply don't outsource RAG evaluation at all.
More from coding & agent
- 7 Mistakes Economists Make With Claude Skills: Not Building Their Own and Making Too Few — aniketapanjwani · 2026-10-23
- MIT Sloan research: AI speeds up coding but delivery still bottlenecks on testing — sven_ai · 2026-10-23
- Agents logging into bank accounts violate most banks' terms, even with user consent — pstAsiatech · 2026-10-23
- Codex desktop shipped with no editor: 4-person team bet everything on agents — sujingshen · 2026-10-23
- Under matched token budgets, most gains of web agent online augmentation vanish — hllo_wrld · 2026-10-23
- First company officially incorporated by an AI agent via Stripe's Atlas MCP — jeff_weinstein · 2026-10-23