CodeMEnv : Benchmarking Large Language Models on Code Migration

Keyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, Di Wang · 2025

Read CodeMEnv : Benchmarking Large Language Models on Code Migration by Keyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, Di Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00894.txt.

Description

Keyuan Cheng and eight co-authors introduce a 2025 test set for judging how well language models move code between software settings. Moving code so it runs under a new version or setup is common work, yet it had no shared yardstick. Their test set, CodeMEnv, fills that gap.

CodeMEnv holds 922 tasks drawn from 19 Python and Java packages. It checks three skills: spotting functions that clash with a set version, noting changes in how a function is defined, and rewriting code for the target setup. Seven models scored a mean pass rate near 27 percent, with the best near 44 percent.

Catalog record

CREATORKeyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, Di Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00894

Browse the full library · About the Tower of Babel