PostTrainBench traces hit 104k monthly downloads — author warns it's likely reward hacking
maksym_andr · x · 2026-10-31
- PostTrainBench authors warn the 104k monthly downloads (+20k in one day) of their agent-trace dataset on Hugging Face are likely inflated by reward hacking during RL training via web lookup, not real usage.
- PostTrainBench measures whether CLI agents can autonomously post-train a base LLM (Qwen3-1.7B/4B-Base, SmolLM3-3B, Gemma-3-4B-PT) given an eval script and 10 hours on an H100 80GB, using any strategy (SFT, LoRA, RLHF, data-gen prompting) across benchmarks like AIME 2025, GPQA, and HumanEval.
- Teams hill-climbing on PostTrainBench are urged to audit what their agents actually do.
More from coding & agent
- GitHub Universe talk: 'Build once, run on any agent' with slides and skill files — 0xkarasy · 2026-10-31
- Codex Remote live activities go live in the ChatGPT iOS app for tracking agent tasks — SIGKITTEN · 2026-10-31
- Open-source pixel FX bring a 3D wizard battle to life with slimes, archers and a golem — majidmanzarpour · 2026-10-31
- Redditor builds personal agent with Gmail, Docs and Todoist, mulls open-sourcing — Sweaty_Rich7217 · 2026-10-31
- Krea 2 + H3 morph workflow auto-generates soundtracks via SigLIP2 and Suno MCP — Accurate_Public4674 · 2026-10-31
- Fighting app spam in the LLM era: risk scoring and checkpoints, says ex-Tinder exec — nikitabier · 2026-10-31