HuggingFace open-sources live agent RL training: LoRA adapters boost coding harnesses

Hugging Face has open-sourced and livestreamed a complete agent reinforcement learning training protocol: three open models — LiquidAI's LFM2.5-24B-A2B (MoE, 2B activated parameters), LFM3-1.7B, and TwiL-8B — ran unmodified inside three real coding agent harnesses (hermes, pi, opencode) while being trained with LoRA adapters. The training environments and reward hacking defenses are fully open-sourced. Results are striking both in effectiveness and cost: after only 20 smoke-test training steps, LFM3-1.7B gained 9.4 ± 6.3 points pass@1 on the opencode harness across 150 held-out tasks never seen in training (69%→79%, 95% CI, paired against baseline), and the full evaluation of 1,350 agent sessions cost only about $5.

Confirmed

• The training uses a compute-efficient architecture: one vLLM per model, one trainer per harness, each attaching a LoRA adapter to shared weights; LFM3-1.7B's three trainers share a single vLLM instance
• The reward is designed as "task validator score + 0.1 × (1 − calls/15)", applied only to correct rollouts. Ablations show a coefficient of 0.3 hurts accuracy, while 0.1 leaves accuracy unchanged and cuts tool calls by 11–17%
• To address the old environments' bias toward data analysis tasks, new RL environments were synthesized by seeding from public agentic trajectories on HF Hub: trajectories are converted into tasks only if they contain verifiable units of work, yielding four new environment types — Repair (fail first, then fix), Build, Extend, and QA — with difficulty stratification via LFM2.5-24B self-evaluation
• The safeagents anti-cheating mechanism was incorporated during training to counter reward hacking; the related envs are fully open-sourced

Why it matters

• It demonstrates that agent capabilities of small models can be significantly improved inside real, unmodified agent harnesses with minimal compute (single GPU, LoRA, few training steps), with evaluation costs driven down to single-digit dollars
• The environment synthesis method (trajectories → verifiable tasks) and reward ablation findings give the community a reusable agent RL recipe, and full open-sourcing lowers the barrier to reproduction

2026-11-06 ~ 2026-11-06 · 8 related posts

• [source] Livestreamed RL training of LoRA harness adapters for Hermes, pi and opencode, fully open-sourced — yb2698 · 2026-11-06
• [source] Hugging Face livestreams open RL training: trajectory-built envs, safeagents, LoRA evals — yb2698 · 2026-11-06
• Hugging Face synthesizes RL envs from open trajectories: Repair, Build, Extend, QA — yb2698 · 2026-11-06
• Synthesizing RL Envs from Open Trajectories: LFM2.5-24B Self-Scores Task Difficulty — yb2698 · 2026-11-06
• Reward Shaping Ablation: 0.1 Bonus Cuts Tool Calls 11-17% at No Accuracy Cost — yb2698 · 2026-11-06
• Evaluating 1,350 Agent Sessions on One GPU for $5 via Shared LoRA Adapters — yb2698 · 2026-11-06
• [source] 20-Step RL on LFM3-1.7B Lifts opencode Pass@1 by 9.4 Points — yb2698 · 2026-11-06
• HF live-streams agent RL: LoRA adapters trained inside real harnesses like opencode — yb2698 · 2026-11-06