The 'Trickle Test': a new eval measuring whether user models disclose information gradually like humans
gharik · x · 2026-09-11
niloofarmire's team shipped a blog, model, and evals, including the 'Trickle Test'—an eval she spent substantial time building to measure the cadence of information disclosure by user models versus humans.
The core insight: a "benchmaxxed" assistant model info-dumps everything in the first turn, while a good user model follows the proper gradual release, like a human would.
It's a concrete, quantifiable lens for evaluating the emerging category of user models.
More from Models
- Microsoft Patches Record 974 Vulnerabilities, Mostly Found by AI — Distinct-Question-16 · 2026-09-11
- DeepSeek V4.1 Flash tops Vals open-weight index at $0.30 per test, with the smallest skills gap — teortaxesTex · 2026-09-11
- Do You Really Need Flagship Models? Dev Argues Medium Effort Covers 80% of Coding — iamaliveix · 2026-09-11
- OpenAI appears to be quietly rolling out managed Agents on its platform — testingcatalog · 2026-09-11
- 30B Open Model OpenResearcher Beats GPT-4.1 on BrowseComp-Plus — TheZachMueller · 2026-09-11
- Surge AI evals: Claude Fable 5.1 leads at 68.7, Gemini 3.8 Flash jumps 12 points on frontier math — echen · 2026-09-11