Gwern's dictum resurfaces: sampling reveals presence of knowledge, never absence
akbirthko · x · 2026-10-07
akbirthko resurfaced Gwern's well-known dictum — "sampling can only reveal presence of knowledge, not absence" — noting that in hindsight, it should have been enough to explain a lot.
The point is a classic in LLM evaluation: a failed generation doesn't prove a model lacks some knowledge or capability; it may just be sampling luck. It's a caution against every "the model can't do X" claim — negative conclusions need heavier sampling or sharper probes, not a single roll of the dice.
More from AGI Musings
- Richard Ngo's hard sci-fi Tinker sketches how AI designs chips for its successors — teortaxesTex · 2026-10-07
- tszzl: Don't be an uncomprehending audience clapping at the AI magic show — Afinetheorem · 2026-10-07
- VraserX: Personal AI agents like Grok Bot, Muse and Dot will soon be everywhere — VraserX · 2026-10-07
- Predicting a 'third thing' RSI: capability leap in ~9 months via automated R&D, short of full FOOM — teortaxesTex · 2026-10-07
- Hinton, Bengio and 20+ experts urge governments to track how much AI research is done by AI — rohanpaul_ai · 2026-10-07
- For Which Generation Will Death Become Optional? — thesaraharminta · 2026-10-07