Evaluating perturbation robustness of generative systems that use COBOL code inputs DOI: XXXXXXX.XXXXXXX Conference: The 1st International Workshop on AI for Software Quality Evaluation: Judgment, Metrics, Benchmarks, and Beyond; April 14, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2018/06
Samuel Ackerman, Wesam Ibraheem, Orna Raz, Marcel Zalmanovici · 2025
Read Evaluating perturbation robustness of generative systems that use COBOL code inputs DOI: XXXXXXX.XXXXXXX Conference: The 1st International Workshop on AI for Software Quality Evaluation: Judgment, Metrics, Benchmarks, and Beyond; April 14, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2018/06 by Samuel Ackerman, Wesam Ibraheem, Orna Raz, Marcel Zalmanovici (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18488.txt.
Description
Samuel Ackerman, Wesam Ibraheem, Orna Raz, and Marcel Zalmanovici build a framework for testing whether AI systems that read COBOL code break under small, meaning-preserving changes. Language model systems are known to wobble when inputs are reworded, and for COBOL the stakes are high: much business-critical software is written in it. Yet the code is proprietary, so models rarely train on it. Their test case is translation between COBOL and Java, though the approach extends to generation and explanation.
The team creates a library of ways to perturb COBOL, from single paragraphs to whole programs, and expands a benchmark with those variants. A system's score then reflects not just whether it works once, but whether it keeps working across harmless changes. The framework targets a blind spot in legacy modernisation projects.
Catalog record
| CREATOR | Samuel Ackerman, Wesam Ibraheem, Orna Raz, Marcel Zalmanovici |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18488