COBOL-to-Python LLM translations pass ~99% of shown examples but 82–91% of hidden tests
Balu0603 · reddit · 2026-10-15
An open-source (MIT) differential-testing study quantifies the gap between "LLM code passes the examples" and "the code is correct," on the nasty-semantics domain of COBOL→Python translation.
Setup: GnuCOBOL runs the original as oracle; 50 programs (30 public, 20 written for the repo); the translator sees only the COBOL source and two example I/O pairs; scoring uses hidden inputs including PICTURE boundary values (max digits, sign flips, rounding ties), with the protocol pre-registered.
Results: plain Python with stdlib only passes 98.9%/100% of shown examples but 91.1%/81.7% of hidden tests; adding a runtime library encoding COBOL semantics lifts hidden tests to 98.9%/98.3%; adding up to 3 repair rounds fed by the first differing field reaches 100%/100%, all fixed in one round.
Key takeaways: prompt examples are a weak oracle—failures were domain semantics (COBOL truncating input to field length, dropping digits on overflow, printing unsigned zero), invisible in two happy-path examples; encoding domain semantics in a library moves almost all failures out of the model's hands. The author also candidly discloses his own runtime bug caused most condition-A failures: check your tools before blaming the model. Limits: small corpus, one model on one date, and program-level A-vs-C differences not statistically significant.
More from Research
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23