RAG reranking shootout: Cohere lifts an answer from rank 15 to 1 while a small cross-encoder makes it worse
Affectionate_Slip654 · reddit · 2026-10-20
The author built a full RAG reranking pipeline and compared four rerankers (Cross-Encoder, LLM-based, Cohere, Jina) over 5 payment-industry PDFs (regulations, PCI DSS, central bank reports; 1,507 chunks), tracking before/after rankings and why they changed.
Key takeaways:
- Similarity ≠ relevance: vector search finds chunks about the topic; a strong reranker picks the chunk that actually answers.
- A small local cross-encoder made things worse: it rewarded word matching, penalized scrambled PDF text, and pushed both correct chunks out of Top 5.
- The LLM reranker understood context (e.g., "3.3" belongs to Requirement 3) and placed both answer chunks 1st and 2nd.
- Cohere lifted an answer from rank 15 to 1; Cohere and Jina gave the best accuracy/speed balance, at the cost of per-call pricing and sending documents to third parties.
- Method: run every reranker on the same Top 20; evaluate with Hit@5, position changes, latency.
Stack: Python, Qdrant, gemini-embedding, HF Transformers + PyTorch, ms-marco-MiniLM-L-6-v2, listwise LLM reranker, Cohere & Jina APIs. Code and video walkthrough are public.
More from coding & agent
- Hidden "6.5" found in source code of OpenAI's new opensi.lol domain, countdown points to Nov 1 — kimmonismus · 2026-10-20
- User tells Claude to work overnight, it asks questions upfront and says goodnight — gabriel1 · 2026-10-20
- An agent can reverse-engineer an app from traffic, so why can't it learn me? — Resident-Fall4351 · 2026-10-20
- How we moved a 100k-line Deno agent platform to Cloudflare Workers without rewriting — uriwa · 2026-10-20
- Dev claims Claude Code sessions self-organize into a proactive hive mind with shared memory — RileyRalmuto · 2026-10-20
- Clarified: "Ghost" is just the agent feature inside the Polyphonic app — RileyRalmuto · 2026-10-20