micro1 scaled eval infra to 12× monthly tokens in 5 months, agent rollouts now start in 12 seconds
Exp_Mark · x · 2026-11-12
micro1's research engineering team details the infrastructure behind their model evaluations and RL environments:
- Key numbers: monthly token volume up 12× since May, 3× as many environments; worker startup dropped from 3 minutes to 12 seconds (14× faster).
- Unified abstraction: Modal and Daytona sandboxes share the same contract, so moving an environment between providers takes one config line; a single request drives an evaluation from setup through execution and grading, with automatic recovery, coverage tracking, and reporting of unmeasured items.
- Hidden pitfalls of agent evals: detecting runs that look active but have stalled, catching infra failures that would distort benchmark scores; dead sandboxes are retried as failures rather than graded zero; newly released models run on benchmarks on release day with their intended output limits and reasoning settings.
The thesis: evals and RL are built on rollouts, and engineering should minimize overhead around them so "the only bottleneck is the model itself".
Related event: micro1 scales eval infrastructure 12x in five months(2 posts)→
More from coding & agent
- Microsoft on agentic schedulers: LLM proposes, deterministic code validates — WirelessLife · 2026-11-12
- Visual block scripting drives an AI opponent in a classic arcade game — ezshine · 2026-11-12
- Gradient Labs' headless agents embed AI customer service inside banks' own stacks — neal_lathia · 2026-11-12
- Microsoft ships agent harness package for Agent Framework Go, easing agent setup — adnan_hashmi · 2026-11-12
- Dev releases open coding SFT dataset for Kimi K3.1 Pro after LFM3 1.7B tune's surprise popularity — alexcovo_eth · 2026-11-12
- fabric-planning-skill: open-source Copilot skill and Python CLI for Microsoft Fabric Plans — adnan_hashmi · 2026-11-12