kapa.ai Benchmarks Jev as a Reranker: 0.003 Behind Voyage rerank-3, Beats Cohere and Qwen
srnsnemil · reddit · 2026-10-20
kapa.ai (YC S23), which runs millions of retrieval queries monthly, compared rerankers on their Company Knowledge Bench — 1,000 test cases built from real production queries over knowledge bases containing product docs, Slack threads, tickets and code.
Setup: GPT-6 Luna decomposes each query into 3 sub-queries → hybrid retrieval (top 30 each) → reciprocal rank fusion → top 50 candidates to each reranker, scored on whether the top 10 chunks satisfy labeled criteria. Jev isn't a dedicated reranker but a general decision model; they tested binary (Noul) and graded (Score) conversions.
Findings
- Jev 1.14 came within 0.003 of first place, beating every Cohere model and Qwen3-Reranker-8B
- Voyage rerank-3 still won while also being the cheapest and fastest
- Results diverge from turbopuffer's benchmark (Jev won) and DevX Labs (12th), showing reranker rankings are highly domain-dependent
The benchmark targets messy real-world company knowledge — stale pages, contradictory facts, multi-answer queries — closer to production traffic than academic sets.
Related event: Jev Proves a Surprisingly Strong Reranker in kapa.ai Benchmarks(2 posts)→
More from Models
- ChatGPT web can actually listen to audio, but the same model can't via API or Codex — Possible_Statement84 · 2026-10-23
- MiniMax M3.1 Flash free for 3 days on 0G, with 1M context and 512K output — MiniMax_AI · 2026-10-23
- TLA+ expert corrects Grok: TLC only exhausts finite state spaces, not all behaviors — tianyin_xu · 2026-10-23
- Open LLM cost calculator shows Arabic tokenization and 6x price gaps across models — Moham-saed · 2026-10-23
- OpenAI blocks manual reset with 13% usage left: 'no reset needed right now' — AccomplishedSugar490 · 2026-10-23
- Meta's 816B Muse Spark Open scores 39, losing to 180B Qwen3.8-Flash-Next at 40 — BTA_Labs · 2026-10-23