Prompts are just keyholes into training distribution, cannot invent missing knowledge
gerardsans · x · 2026-08-21
The author argues that prompting only works if the training corpus covers the task, serving as a "keyhole" into the training distribution. If the solution doesn't exist within that distribution, no amount of retries or "better prompting" will invent it. Consequently, regular users without deep math and model understanding cannot advance science using AI alone.
More from AGI Musings
- Cheating is ubiquitous in academia, plagiarism is low-IQ version — kevinnbass · 2026-08-21
- Naval: Frontier Labs incentivized to be fearful to justify staying closed, while China stays open — naval · 2026-08-21
- AI Consciousness and Anthropomorphism: Generative Models Are Just Statistical Interpolations — gerardsans · 2026-08-21
- Jevons Paradox in AI: Personal compute will never be enough — MannyKayy · 2026-08-21
- New paper calls for total opposition to AI, claiming it will end the math profession — pickover · 2026-08-21
- Poll shows Americans distrust AI developers, but battle for minds remains open — NinaDSchick · 2026-08-21