Cross-encoder reranking loses to a stricter cosine threshold on 2 of 3 semantic cache benchmarks
Reasonable_Royal_621 · reddit · 2026-10-19
The author shares a counterintuitive empirical result from CacheVerifier: sending borderline semantic cache matches (cosine 0.80–0.97) to a cross encoder was expected to beat raw cosine, but lost on 2 of 3 benchmarks.
- SearchQueries (86k borderline pairs): top 5% by score — reuse was correct 73.2% of the time ranked by cosine vs 67.5% by the cross encoder; same story on Quora. Only LmArena, where the stock model is already decent, favored the cross encoder.
- Fine-tuning fixes it: a verifier fine-tuned per dataset beat a plain threshold at all 18 tested points.
- Debugging note: they had this backwards for a while because their eval scored the two sides on different requests.
Takeaway: before shipping a reranker, compare it against just a stricter cosine cutoff. Repo: github.com/imxinchengyou/CacheVerifier
More from Infra
- "Slowest pretrain I've ever supervised": researcher's one-line lament — cephaloform · 2026-10-19
- Sipeed's MaixStation challenges DGX Spark: 128GB LPDDR5X at 40W, one-third the size — AIFlow_ML · 2026-10-19
- ReaDy Uses Recurrent States as Dynamic Context for Speculative Decoding in Hybrid Linear Attention LLMs — Bohan Sun · 2026-10-19
- TRELLIS.2 image-to-3D running on AMD RX 9060 XT with ROCm: fixes, workflows, MIT swaps — ova-mojica · 2026-10-19
- Celesto Claims CI Runner With 96% Lower Per-Minute Cost Than GitHub Actions — aniketmaurya · 2026-10-19
- Huawei's Atlas 960E superpod links 4,096 NPUs with near-packaged optics — bookwormengr · 2026-10-19