Beam-501B agent demo: solo beats 5-agent team on ProgramBench rebuild task
ben_burtenshaw · x · 2026-10-16
Hugging Face's benburtenshaw benchmarked Reflection's Beam-501B-A23B with the Pi coding agent on ProgramBench (rebuilding programs from binaries, no source) under three orchestrations: solo, a 5-peer team sharing a git repo, and lead + async subagents — all with the same token budget.
The score-vs-latency chart shows the solo agent (1M context tokens) reaching 0.43 hidden-test pass fraction, edging out the multi-agent setups (0.30–0.38), though agent teams hit usable scores earlier on some tasks. Caveats: 4-task subset, no confidence intervals, 8M API-token cap per episode.
More from coding & agent
- Deep Research API showdown: OpenAI, Perplexity, Parallel, Tavily, Gemini, Valyu compared — bar_raiser333 · 2026-10-16
- Geoffrey Huntley open-sources Overalls, a self-hosted AI developer platform under MIT — kieranklaassen · 2026-10-16
- Dev shows 199.7bn tokens over 200 days on OpenAI Codex, peak 8.6bn/day — Dimillian · 2026-10-16
- Ask Claude to open its changes in nvim and scroll to the exact line for review — max__drake · 2026-10-16
- Proof CEO on x401: KYC-compatible agent identity using NIST IAL2 verifiable credentials, composing with x402 payments — csuwildcat · 2026-10-16
- Pydantic's HTTPX2 adds native WebSocket and SSE support — samuelcolvin · 2026-10-16