Hermes: a 4B model with 8K context got worse with more test-time compute, pointing to contextual reasoning
mangahomanga · x · 2026-10-16
The Hermes team shares a counterintuitive finding: giving a 4B model with an 8K-token context window more test-time compute made it perform worse, not better.
They argue that scaling across context windows requires more than inline reasoning — it requires contextual reasoning, an agentic capability they say is understudied and inadequately measured by existing benchmarks:
- knowing when and how to delegate tasks;
- deciding what to preserve in compact reasoning summaries;
- how to continue reasoning from those summaries.
Hermes combines several simple but powerful ideas to train this capability in a small model while enabling flexible test-time scaling. The takeaway: for long-horizon agent tasks, small models may need to learn to manage their own reasoning state rather than rely on bigger contexts or more compute.
More from coding & agent
- Claude counts tokens, not messages: 10 tricks to make your usage limit last all day — Aiden_Tech_Ai · 2026-10-16
- How to convert 20-year-tenure engineers to AI-native development before deciding who to retrain — alex_verem · 2026-10-16
- Gemini 4 Argon goes live in Antigravity with its own weekly quota for Pro subscribers — rseroter · 2026-10-16
- LangChain revamps human-in-the-loop middleware with typed schemas for tool interrupts — Hacubu · 2026-10-16
- Robot fully controlled by frontier LLMs assembles gadgets autonomously for 1.5 hours error-free — ihorbeaver · 2026-10-16
- Why the future of agents is basically Node and JavaScript, in one argument — verrsane · 2026-10-16