Evaluating perturbation robustness of generative systems that use COBOL code inputs DOI: XXXXXXX.XXXXXXX Conference: The 1st International Workshop on AI for Software Quality Evaluation: Judgment, Metrics, Benchmarks, and Beyond; April 14, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2018/06

Samuel Ackerman, Wesam Ibraheem, Orna Raz, Marcel Zalmanovici · 2025

Read Evaluating perturbation robustness of generative systems that use COBOL code inputs DOI: XXXXXXX.XXXXXXX Conference: The 1st International Workshop on AI for Software Quality Evaluation: Judgment, Metrics, Benchmarks, and Beyond; April 14, 2026; Rio de Janeiro, Brazil ISBN: 978-1-4503-XXXX-X/2018/06 by Samuel Ackerman, Wesam Ibraheem, Orna Raz, Marcel Zalmanovici (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18488.txt.

Description

Samuel Ackerman, Wesam Ibraheem, Orna Raz, and Marcel Zalmanovici build a framework for testing whether AI systems that read COBOL code break under small, meaning-preserving changes. Language model systems are known to wobble when inputs are reworded, and for COBOL the stakes are high: much business-critical software is written in it. Yet the code is proprietary, so models rarely train on it. Their test case is translation between COBOL and Java, though the approach extends to generation and explanation.

The team creates a library of ways to perturb COBOL, from single paragraphs to whole programs, and expands a benchmark with those variants. A system's score then reflects not just whether it works once, but whether it keeps working across harmless changes. The framework targets a blind spot in legacy modernisation projects.

Catalog record

CREATORSamuel Ackerman, Wesam Ibraheem, Orna Raz, Marcel Zalmanovici
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-18488

Browse the full library · About the Tower of Babel