LessWrong Essay Introduces LM Psychology: Just Talking to an AI Still Beats Fancy Evals
jankulveit · x · 2026-10-21
jankulveit and Raymond Douglas published a long-form LessWrong essay, "A brief(ish) introduction to LM psychology," laying out the discipline of understanding language models by talking to them directly.
Key points:
- The post includes practical advice for doing LM psychology well, a reading list, and open research questions the authors are excited to see explored.
- Their contrarian take: despite years of work on evaluations and interpretability, in many ways we have yet to outdo "just talk to an AI and see what you think of it."
- They argue neither this nor other current techniques will last forever, so it's worth doing the naive method well.
Tied to the same thread: ACS Research now offers a free platform with Llama 3.1 8B/405B base models, lowering the barrier for individual researchers in this space.
More from Research
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- Anthropic: MLP Neurons Are a Stronger Basis for Interpretability Than Assumed — omarsar0 · 2026-10-21
- Microsoft's Self-HPO: Coding Agent Redesigns Its Own Harness, +7 Points on SWE-bench — mark_k · 2026-10-21
- Tim Dettmers: Academia is about to have a renaissance — the best AI work of the next decade happens in university labs — StasBekman · 2026-10-21
- Stanford and Laude release Atlantic-CC, a 17T-token model-filtered Common Crawl for open LLMs — Diyi_Yang · 2026-10-21
- AI saves scientists 7 hours a week, but validation is the new bottleneck — AnneliesGamble · 2026-10-21