Benchmarking and Revisiting Code Generation Assessment: A Mutation-Based Approach

Longtian Wang, Tianlin Li, Xiaofei Xie, Yuhan Zhi, Jian Wang, Chao Shen · 2025

Read Benchmarking and Revisiting Code Generation Assessment: A Mutation-Based Approach by Longtian Wang, Tianlin Li, Xiaofei Xie, Yuhan Zhi, Jian Wang, Chao Shen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06880.txt.

Description

Longtian Wang and five co-authors probe a weakness in how code-writing language models are graded, in a 2025 paper. Standard benchmarks give each programming problem one fixed prompt, but real users phrase the same problem many ways, typos included. A model's score can swing with the wording, so a single prompt may misstate its skill.

The team designed ten mutation strategies that perturb prompts in realistic ways and three new metrics to measure the fallout. They generated 12,834 prompt variants and tested five popular code models. Scores on the mutated prompts differed significantly from the clean benchmark results. The authors argue that future benchmarks need this kind of robustness testing built in.

Catalog record

CREATORLongtian Wang, Tianlin Li, Xiaofei Xie, Yuhan Zhi, Jian Wang, Chao Shen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06880

Browse the full library · About the Tower of Babel