Ultramarine blog reveals RL reward formulas penalizing token waste and broken tool calls
andrew_n_carr · x · 2026-10-15
A summary of the reward-shaping design in Paper Instruction's Ultramarine launch blog targets two agent-training pain points: excessive token generation and invalid/broken tool calls.
The shared formulas show:
- Each invalid tool call incurs a λ penalty, capped in total at ctool, so models aren't discouraged from exploring entirely;
- A turn factor f(n) fades positive reward linearly from turn n₀ to zero at cap nmax, preventing score-farming by stalling;
- An output-length penalty P(T) activates (gated by Gg) when the group mean reward is positive, combining a proportional over-budget penalty and a tail clip penalty.
The design goal: carefully encourage exploration without letting models get away with breaking things.
More from coding & agent
- Open-Agent-Email: free open-source alternative to AgentMail for AI agents — matchaman11 · 2026-10-15
- AWS Gen AI Weekly Redesign: One-Page Exec Summary, AgentCore Memory Goes Usage-Based — DigitalColmer · 2026-10-15
- Nous Research Launches 50+ Connectors for Hermes Agent, No Account Required — NousResearch · 2026-10-15
- Dev group chat: skip runtime LLM web search, index products via API instead — vxnuaj · 2026-10-15
- Tev-Gym: 1,170 dense-reward RL environments push long-horizon agentic training — RulinShao · 2026-10-15
- Microsoft's AutoEnvScaling lets agents build their own RL environments, lifting Qwen3 Terminal-Bench from 41.1 to 53.3 — omarsar0 · 2026-10-15