Researchers debate whether a code harness, not the base model, is the real unit of comparison
srush_nlp · x · 2026-07-23
The discussion asks whether the real novelty is not just that harnesses help generalization, but that a harness treated like a model changes the comparison itself.
The reply argues:
- The base Transformer does not get tools, so the relevant baseline is harness-like programs vs. the base model.
- On some tasks, a code tool makes an obvious difference.
- On others, the benefit is less clear because chunking and acting on chunks is not unique to LM programs; in principle, model weights could be constructed to do something similar.
- The point being made is therefore about the role of the code/tool harness as part of the system, not just the underlying LM.
Related event: LLM Generalization Debate: Intrinsic Model or Harness Contribution?(11 posts)→
More from coding & agent
- Nitro Modules v1 gets Node.js support: one native module codebase for iOS and macOS — Birchlabs · 2026-10-27
- Google Graph Mining Team Builds Trillion-Edge FarAway with Gemini 4 Argon Agents in 5 Hours — rseroter · 2026-10-27
- Four Gemini agents beat Minecraft autonomously in 27 hours, writing 2,783 programs — rohanpaul_ai · 2026-10-27
- Planning a Master-Agent Architecture: LangGraph vs Microsoft Agent Framework — DonXendon · 2026-10-27
- Vibe-coding a better Matter smart home setup in 10 minutes after iPhone Home app timeouts — blixt · 2026-10-27
- Score centering lands in TRL: default-on for async GRPO, stabilizes RL under mismatch — _lewtun · 2026-10-27