Builder replaces value model with an LLM call, routes estimator outputs to the trainer
cephaloform · x · 2026-08-18
A first-hand engineering update on an agent training system. The previous reward setup relied on a naive sliding window plus guideline-based turn judgments with EMA-based advantages, which accidentally clipped tasks. Now the model itself decides window size using tools, fixing that issue.
Next up: routing the value estimator's outputs to the trainer so it co-evolves with the agent better than RAG with timestamped memories. The author jokes that they once quipped value models would be replaced by a single LLM call — and now they're actually doing it.
More from coding & agent
- Anthropic reportedly developing Hub Mode for agents — testingcatalog · 2026-08-18
- Handling memory overflow in single-file agent logs — Interesting-Year-418 · 2026-08-18
- KAISEN AI uses genetic algorithms and local LLMs to optimize code — andreabarbato · 2026-08-18
- Compound Engineering Update: New Skills and Windows Support — every · 2026-08-18
- Agent coding tips: run /simplify, then fresh-context review of the diff — lucasmeijer · 2026-08-18
- Developer claims Go is miles ahead for AI coding agents — dosco · 2026-08-18