Benchmarking 4 RAG Rerankers: Cohere Lifted the Answer from Rank 15 to 1
Affectionate_Slip654 · reddit · 2026-10-14
The author built a complete RAG reranking pipeline (retrieve Top 20 → rerank → Top 5 → LLM) and benchmarked four rerankers—local Cross-Encoder, LLM-based reranking, Cohere Rerank, and Jina Reranker—on 5 payment-industry PDFs (PCI DSS, central bank reports, 1,507 chunks), with full code and a video walkthrough.
Key findings:
- Similarity ≠ relevance: vector search finds chunks about the topic, not necessarily ones that answer the question
- A small cross-encoder can hurt: it rewards word matching and pushed both correct chunks out of the Top 5
- LLM reranking understood semantics (e.g., that "3.3" belongs to Requirement 3), placing both answer chunks 1st and 2nd
- Cohere lifted one answer chunk from 15th to 1st, turning a miss into a correct, cited answer
- Cohere and Jina offered the best accuracy/speed balance here, with trade-offs in per-call cost and sending docs to third parties
- Evaluate with before/after positions, Hit@5, and latency, always on the same Top 20 set
Stack: Python, Qdrant, gemini-embedding-2-preview (3,072-dim), ms-marco-MiniLM-L-6-v2, Cohere/Jina APIs. Code layout: retriever.py → rerankers.py → generate.py → compare.py.
More from coding & agent
- Codex autocompaction gets stuck: one steer and 10 million tokens later it's still parroting stale state — Aryvyo · 2026-10-14
- Agentic engineering vs AI engineering: Hugo Bowne-Anderson's clean split — Al_Grigor · 2026-10-14
- PostHog to devtools: stop asking 'how to add AI,' ask 'what's the agent-first version' — edwin · 2026-10-14
- Devin SWE-2 jumps to #3 on attention-to-detail coding test, tying Claude Sonnet 5.5 — silasalberti · 2026-10-14
- MCP hits ~500M monthly downloads but stays free — the money goes to key-management security firms — FinanceYF5 · 2026-10-14
- One-shot flying dragon: Gemini 4 Argon with Blender MCP and Tripo 3D builds a pure .blend scene — sidahuj · 2026-10-14