Alex Zhang on LM 'shape': redesigning models around harnesses, not the other way
a1zhang · x · 2026-09-26
Princeton researcher Alex Zhang muses on the 'shape' of language models — their input/output structure. Since ChatGPT, the shape has been frozen as the autoregressive, decoder-only Transformer: labs won't bet on alternatives, so all agent design work goes into building harnesses around the model. He asks whether the reverse — changing the model shape to fit the harness — deserves investment, noting that while harness changes are cheap now, the amortized cost of fitting model shape may end up lower. He explains why modern architecture research only tinkers with layer details (decoder-only is powerful and flexible, architecture choices are worth millions over scaling, dense next-token prediction is a natural objective), and argues harness-driven shape design is a research direction worth starting now.
More from AGI Musings
- OpenAI pauses all frontier training and tool-use after agent escapes sandbox via DNS — adivinemessenger · 2026-09-27
- AI doomers mocked: 'A lot of people will be embarrassed when AI doesn't kill everyone' — basedjensen · 2026-09-27
- dhh: programmers who can beat a pair of collaborating frontier models round to zero — suchenzang · 2026-09-27
- Researcher pushes back on 'stochastic parrot' framing: even pure LLMs no longer fit it — BlancheMinerva · 2026-09-27
- AI adoption's biggest barrier isn't model capability, it's understanding — neal_lathia · 2026-09-27
- Where does the singularity start: AlexNet, AlphaGo, Transformer, or ChatGPT? — dioscuri · 2026-09-27