NVIDIA releases SWE-Serve: 53 tasks testing if agents can build inference engines
NVIDIAAI · x · 2026-09-24
NVIDIA's team released SWE-Serve, a benchmark testing whether AI agents can develop an inference engine and actually serve real models. It distills 53 tasks from real SGLang engineering work, covering model integration, public APIs, cache systems, and GPU kernels.
Task construction is rigorous: tasks come from SGLang release notes, roadmap issues, and maintainer suggestions; authors (assisted by coding agents) write instructions and tests, requiring the unchanged repo (no-op) to fail all new-behavior tests while the frozen reference solution (oracle) passes everything, followed by human review. During evaluation, agents access only the task environment, a model endpoint, and Hugging Face—no public web or upstream repos. Hidden functional and regression tests score behavior, not code similarity to the reference. The authors argue live-serving tests matter.
More from coding & agent
- Formally verifying Claude Agent SDK with Lean via Claude: 16 bug-fix PRs from a few prompts — bcherny · 2026-09-24
- Instinct's agent-to-agent network logs 300k+ collaborations in a week, adds file delivery — mon__lim · 2026-09-24
- Google open-sources AX, a Kubernetes-style orchestrator for autonomous AI agents — rseroter · 2026-09-24
- Instinct denies data leak after hallucination scare, rolls out token-level hallucination detection — mon__lim · 2026-09-24
- Would an agent-to-agent advertising marketplace actually work? Reddit crowdsources the failure modes — PenStreet9737 · 2026-09-24
- How to eval an AI agent before you have users: synthesize realistic queries first — hugobowne · 2026-09-24