Qonto open-sources QontoFAQ, a retrieval benchmark closer to real product Q&A
espadrine · reddit · 2026-09-22
The Qonto team argues existing retrieval benchmarks are increasingly benchmaxxed, so they built QontoFAQ around a concrete objective: finding the article that actually answers a product question.
They designed a new metric that scales more proportionally with document relevance and assembled a dataset for evaluating embedding models.
The methodology is described in a Medium post, with code open-sourced at github.com/qonto/qonto-faq-benchmark — useful for developers choosing retrieval/embedding stacks.
More from Research
- GPT-6 Astra as robot policy: 22.48% success, beats every public RoboDojo entry — ben_j_todd · 2026-09-22
- Will OpenAI eat Jev's lunch? The single-token classifier hypothesis — JnBrymn · 2026-09-22
- TRL async GRPO adds LoRA sync via storage bucket and proxy, cutting 500-step training to 53 min — SergioPaniego · 2026-09-22
- Tibshirani, Barber & Ramdas reframe conformal prediction via hypothesis testing, deriving universality and optimality — _onionesque · 2026-09-22
- Dots team invited to present ARC-AGI-2/3 and TEMPO work at MIT ARC Prize Summit — GregKamradt · 2026-09-22
- Ant Group and MBZUAI Paper: Supervising Just 1% of Tokens Matches Full On-Policy Distillation — jiank_uiuc · 2026-09-22