Kaggle and IBM launch ScarfBench: 204 tasks testing coding agents on enterprise Java migration
gabriberton · x · 2026-11-02
Kaggle and IBM Research launched ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark testing whether coding agents can migrate enterprise Java apps across frameworks like Jakarta EE, Quarkus, and Spring while preserving expected behavior.
- Framework migration requires coordinated changes across dependency injection, persistence, build systems, and deployment — agents easily produce migrations that compile but fail at deploy time, which is exactly what ScarfBench stresses.
- It evaluates each model with its own harness over 204 migration tasks; the post's mention of Gemini 4 Argon's results is cut off, full leaderboard at the link.
A new benchmark worth running for anyone doing agent evals or enterprise code migration.
More from coding & agent
- Comic nails the agent economy: humans just approving approvals, forever — sierracatalina · 2026-11-02
- Back up your Grok bot daily to a private repo: 20+ agents of skills and memories — 0xkarasy · 2026-11-02
- Revolut cuts AI costs 10x by self-hosting GLM on 56 B200s, boosting lead time 70% — SumitGup · 2026-11-02
- New study examines how developers actually supervise autonomous coding agents — burkov · 2026-11-02
- mitsuhiko's agent-built readline rewrite sparks GPL-to-MIT license laundering debate — mitsuhiko · 2026-11-02
- 4B coding model generates Three.js scenes with GLB export for Godot game assets — ConfidentDinner6648 · 2026-11-02