Google's T2J-Bench shows coding agents score just 0.676 migrating PyTorch repos to JAX
CShorten30 · x · 2026-10-20
Google researchers introduced T2J-Bench, testing whether coding agents can migrate entire PyTorch repositories to native, torch-free JAX/Flax while preserving behavior:
- Scale: 63 whole-repository tasks across 18 model families, covering foundation-model training (SFT/DPO/PPO) and serving
- Evaluation: agents get source code plus a raw checkpoint; an offline verifier compares outputs against frozen PyTorch references on spec, numeric (params/loss/gradients), and end-to-end behavioral levels, yielding a continuous reward
- Findings: the best off-the-shelf setup reaches only 0.676 reward vs a 0.980 JAX oracle — getting code to run is only the beginning
- TwinHarness: a reference-grounded migration loop lifts reward from 0.649 to 0.723 (+11.4% relative) with the same agent and model
Code is open-sourced on GitHub under Apache-2.0.
More from coding & agent
- Acellera runs an unsupervised virtual lab of AI scientists on a real IL-23 molecular discovery program — gdefabritiis · 2026-10-21
- Pydantic author experiments with JS/TS and React support in the Monty sandbox — samuelcolvin · 2026-10-21
- Supabase lets coding agents query your logs to catch errors and slow queries — dshukertjr · 2026-10-21
- $200-tier user says Codex burned 25K credits in days, demands answers from OpenAI — alexcovo_eth · 2026-10-21
- Agent memory debate: save memories whole, build retrieval paths for cold storage — mimi10v3 · 2026-10-21
- tsrs: agent-built Rust port of TypeScript runs 8x faster with 70% less memory — irvinebroque · 2026-10-21