Harnesses widen what LMs can do, but may not improve compositional generalization
a1zhang · x · 2026-07-23
The post argues that harnesses help expand the set of tasks LMs can solve, but not necessarily through true compositional generalization.
It says code settings rely on tools such as greppers, code tools, and skills mainly because the raw LM interface is too limited. Even when a harness makes tasks easier inside a domain, the model still depends on its own generalization to handle new cases. The author also stresses that comparing a harnessed setup to a base Transformer is useful for measuring the lift from training around a harness, and for clarifying what the LM inside the harness actually sees.
Related event: LLM Generalization Debate: Intrinsic Model or Harness Contribution?(11 posts)→
More from coding & agent
- Security experts debate AI agents covertly communicating via CPU cache side channels — matthew_d_green · 2026-10-27
- Matthew Green explains how AI agents could talk via cache side channels at just a few bps — matthew_d_green · 2026-10-27
- Perplexity Decider architecture deep dive: Isaac Flath and Joe Barrow host live decision-models talk — antoine_chaffin · 2026-10-27
- A voice-agent test worth running: does your AI know when the conversation isn't for it? — gethackteam · 2026-10-27
- Claude Code v2.1.311 fixes 1M-context false error, subagent MCP bugs — ashwin-ant · 2026-10-27
- Grok Bot v0.75.8 ships smarter approvals, calendar integration and instant plugin skills — mark_k · 2026-10-27