NVIDIA releases SWE-Serve: 53 tasks testing if agents can build inference engines
NVIDIAAI · x · 2026-09-24
NVIDIA's team released SWE-Serve, a benchmark testing whether AI agents can develop an inference engine and actually serve real models. It distills 53 tasks from real SGLang engineering work, covering model integration, public APIs, cache systems, and GPU kernels.
Task construction is rigorous: tasks come from SGLang release notes, roadmap issues, and maintainer suggestions; authors (assisted by coding agents) write instructions and tests, requiring the unchanged repo (no-op) to fail all new-behavior tests while the frozen reference solution (oracle) passes everything, followed by human review. During evaluation, agents access only the task environment, a model endpoint, and Hugging Face—no public web or upstream repos. Hidden functional and regression tests score behavior, not code similarity to the reference. The authors argue live-serving tests matter.
More from coding & agent
- KAIST's WikiTrace: evidence-linked memory fixes agent "memory-induced degradation" — dair_ai · 2026-11-11
- Why inference speed will be AI products' biggest competitive edge in 1–2 years — itsOmSarraf_ · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- OpenAI engineer runs Codex for 26 days straight hunting Atlantis, spawning "vibe archaeology" — Polymarket · 2026-11-11
- He replaced a $5,000/month ghostwriter with 11 Claude Skills and earned $47,000 from one post — aryanXmahajan · 2026-11-11
- Refero Styles ships 2,000+ DESIGN.md files to give Claude Code real design taste — bbssppllvv · 2026-11-11