Beyond BLEU: A Semantic Evaluation Method for Code Translation

Julius Näumann Sven Keidel Amir Molzam Sharifloo Mira Mezini · 2026

Read Beyond BLEU: A Semantic Evaluation Method for Code Translation by Julius Näumann Sven Keidel Amir Molzam Sharifloo Mira Mezini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05282.txt.

Description

Julius Näumann and three co-authors propose judging code translation by whether the translated program works, not by how similar its text looks to a reference. Metrics such as BLEU score surface likeness and ignore program meaning, yet translation is a core language model skill. The authors borrow standard compiler testing methods and apply them to code conversion quality.

Their meaning-based score is the share of translations that run to correct results, shown on models fine-tuned for binary lifting, which turns binaries into higher-level code. Language-model approaches beat rule-based tools by far on that score. Meanwhile BLEU barely tracked the meaning score, showing text metrics fail to predict whether converted code works.

Catalog record

CREATORJulius Näumann Sven Keidel Amir Molzam Sharifloo Mira Mezini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05282

Browse the full library · About the Tower of Babel