Researchers debate whether a code harness, not the base model, is the real unit of comparison
srush_nlp · x · 2026-07-23
The discussion asks whether the real novelty is not just that harnesses help generalization, but that a harness treated like a model changes the comparison itself.
The reply argues:
- The base Transformer does not get tools, so the relevant baseline is harness-like programs vs. the base model.
- On some tasks, a code tool makes an obvious difference.
- On others, the benefit is less clear because chunking and acting on chunks is not unique to LM programs; in principle, model weights could be constructed to do something similar.
- The point being made is therefore about the role of the code/tool harness as part of the system, not just the underlying LM.
Related event: LLM Generalization Debate: Intrinsic Model or Harness Contribution?(11 posts)→
More from coding & agent
- One-prompt game dev: GPT-6 Astra plans and codes while Hyper3D Rodin auto-generates every 3D asset — CodeByPoonam · 2026-09-11
- Agentic Trading Lab: An Open-Source Python Library for LLM-Powered Trading Agents — mdancho84 · 2026-09-11
- Recreating a Minimax song with YuE2 via a Hermes agent in pure CLI on a 16GB GPU — wzwowzw0002 · 2026-09-11
- How AI Agents Turn Experience Into Lasting Gains: A Guide to Recursive Self-Improvement — Roger_M_Taylor · 2026-09-11
- Dev torn on Cloudflare Agents SDK: full primitives but vendor lock-in — MikkoH · 2026-09-11
- Recreating a Minimax song with YuE2 via a Hermes agent in pure CLI on a 16GB GPU — wzwowzw0002 · 2026-09-11