UPenn lab shows 'critique and revise' prompting doubles Claude's research ideation output
arjunrajlab · x · 2026-10-16
Arjun Raj's lab at UPenn (with Claude) tested nine prompting methods for generating research ideas across Haiku/Sonnet/Opus 5.5, two cell-biology questions, and 2,160 ideas blind-judged by Opus 5.5 (novel+useful, ≥4/5 both). Key findings:
- Plain prompts scored 0 on the main measure for every model
- Best method, "critique and revise" — showing the model its own default ideas as obvious and asking for replacements — yielded 34/240 qualified ideas; a list-then-avoid variant got 25
- Gains scale with model size: +19 for Opus, +10 for Sonnet, +5 for Haiku
- Only critique-and-revise beat the prior best (borrowing from unrelated fields) by 18 ideas
- Forcing poems or random-word association worked worse than simply asking for non-obvious ideas
No human scientists have rated the ideas yet; report and code are public.
Related event: Asking for Non-Obvious Ideas Boosts LLM Creativity Most, Study Finds(3 posts)→
More from coding & agent
- ThePrimeagen mocks AI coding: infinite tokens still produce an Electron app — seatedro · 2026-10-17
- Generative UI 101: Building Live Interfaces with Agents on Opus 5.5 and Rivals — EXM7777 · 2026-10-17
- MCP server review: gate tool writes on auth/traversal findings, not test score — SamuelFmedeiros · 2026-10-17
- DeepSeek shows rapid vision gains: agent self-checks screenshots to build 3D scene — cedric_chee · 2026-10-17
- He gave an AI agent phone and email access to collect a $500 unpaid invoice — curious_vii · 2026-10-17
- AI:AM transcript: agent lockouts, hidden valence, and formal proofs for Linux kernel bugs — The Cognitive Revolution · 2026-10-17