LoCaL: Countering Surface Bias in Code Evaluation Metrics DOI: XXXXXXX.XXXXXXX CCS: Software and its engineering Functionality CCS: Software and its engineering Software reliability
Simantika Bhattacharjee Dristi, Matthew B. Dwyer · 2025
Read LoCaL: Countering Surface Bias in Code Evaluation Metrics DOI: XXXXXXX.XXXXXXX CCS: Software and its engineering Functionality CCS: Software and its engineering Software reliability by Simantika Bhattacharjee Dristi, Matthew B. Dwyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15397.txt.
Introduction
Simantika Bhattacharjee Dristi and Matthew Dwyer study how code written by models is graded. Grading code by tests is costly, so many tools score new code by how much it looks like a known good answer. Past work showed those scores track real correctness weakly. The causes were guessed at, and fixes were scarce. The pair set out to test the cause directly.
They check four leading tools and find a strong bias toward surface traits over real function. Code that looks alike but acts unlike can fool them, and the reverse can fail them. They build a test set of over three thousand code pairs aimed at those weak spots. The set gives tool makers a hard, fair test.
Catalog record
| CREATOR | Simantika Bhattacharjee Dristi, Matthew B. Dwyer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15397
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LoCaL: Countering Surface Bias in Code Evaluation Metrics DOI: XXXXXXX.XXXXXXX CCS: Software and its engineering Functionality CCS: Software and its engineering Software reliability Simantika Bhattacharjee Dristi, Matthew B. Dwyer Abstract . With the increasing popularity of large language models (LLMs) and LLM-based agents, reliable and effective code evaluation metrics (CEMs) have become crucial for progress across several software engineering tasks. While popular benchmarks often provide test cases to assess the correctness of generated code, crafting and executing test cases is expensive. Reference-based CEMs provide a cheaper alternative by scoring a candidate program based on its functional similarity to a reference. Although prior research has focused on reporting the weak correlation between these CEMs and functional correctness, the causes are only assumed, and plausible solutions remain unexplored. In this work, we critically evaluate four state-of-the-art reference-based CEMs, revealing their strong bias towards surface-level features rather than code functionality. Despite this surface bias , current evaluation datasets for these CEMs rarely include code pairs that are surface-similar yet functionally dissimilar, or functionally similar yet surface-dissimilar. To mitigate this gap, we propose LoCaL ( Lo oks Ca n L ie) , a CEM evaluation benchmark, with 3117 code pairs at both the method and program levels. Each pair is labeled with a functional similarity score and aims to target regions where CEMs are likely to perform poorly. The functional similarity scores are calculated through differential fuzzing, which eliminates the need for predefined test cases and, at the same time, improves the reliability of the scores by executing an order of magnitude more tests than prior work. We find that all four CEMs show significant performance degradation on LoCaL, compared to the baselines. Finally, based on our findings, we draw the implication that exposing CEMs to LoCaL-like data might facilitate the development of metrics that are robust to surface bias . 1. Introduction Automated code generation has emerged as a focal point in agent-driven software engineering (SE), primarily due to its ability to significantly reduce development cost and enhance programming efficiency ( Islam et al. 2024 ; Li et al. 2024 ) . This, in turn, has led to rapid advancements in code generation techniques ( Dong et al. 2023 ; Yin and Neubig 2018 ; Li et al. 2022 ; Mu et al. 2024 ; Siddiq et al. 2024 ) , with tools like ChatGPT ( OpenAI 2025 ) and Github Copilot ( Chen et al. 2021 ) being widely adopted in practice. Code-generation approaches have been developed using code evaluation metrics (CEM) that assess the quality of machine-generated code. Such metrics, though developed for code generation, are equally essential for a broad range of software engineering tasks, including code completion, code translation, code refactoring, code optimization, and automated program repair ( Zhu et al. 2022 ; Pan et al. 2024 ; Fatima et al. 2024 ; Xu 2025 ; Zhao et al. 2024 ) . While prior research in code evaluation has enabled initial progress across a range of agent-driven SE tasks, improvement in CEMs has the potential to drive further advances in these areas. The evaluation of generated code has long relied on test cases to check functional correctness ( Tosi 2024 ; Chen et al. 2022 ; Liu et al. 2023b ) . Reference-based static metrics ( Ren et al. 2020 ; Eghbali and Pradel 2023 ; Zhou et al. 2023 ; Dong et al. 2025 ) are a more recent approach that estimates the functional similarity between a reference and a generated code without execution. These metrics are attractive because they are test-case agnostic and offer faster analysis than execution-based methods ( Jiang et al. 2025 ; Ren et al. 2020 ; Eghbali and Pradel 2023 ; Zhou et al. 2023 ; Dong et al. 2025 ) , but they lack the ground truth of test-based equivalence approaches. To date, there has been little research focused on critically evaluating the reliability of these reference-based static code evaluation metrics. In many SE tasks, the key criteria for assessing code is related to its functional behavior, and to support such tasks, CEMs must capture code semantics rather than surface-level textual or structural similarity ( Evtikhiev et al. 2023 ) . In real-world scenarios, code that appears nearly identical can behave differently, while code that looks different may be functionally identical. For example, as shown in Fig. 1 , code (a) and its mutated variant (b) appear almost alike but diverge drastically in their execution results, while (a) and its optimized variant (c) look substantially different but preserve the functional behavior. These cases highlight a fundamental question that must be asked while evaluating reference-based static CEMs – How reliably do these CEMs capture functional similarity in scenarios where surface-level resemblance and functional behavior diverge significantly? (a) ⬇ def is_palindrome ( text : str ) -> bool : for i in range ( len ( text )): if text [ i ] != text [ len ( text ) - 1 - i ]: return False return True (b) ⬇ def is_palindrome ( text : str ) -> bool : for i in range ( len ( text )): if text [ i ] == text [ len ( text ) - 1 - i ]: return False return True (c) ⬇ def is_palindrome ( input_str : str ) -> bool : return input_str == input_str [::-1] (a, b) : Surface Similarity = 0.99; Functional Similarity = 0.00 (a, c) : Surface Similarity = 0.48; Functional Similarity = 1.00 Figure 1. Examples of surface–semantic divergence in code. Red borders mark how trivial changes in the token-level can flip code semantics entirely. In this paper, we evaluate four popular reference-based code evaluation metrics: CodeBLEU ( Ren et al. 2020 ) ; CrystalBLEU ( Eghbali and Pradel 2023 ) ; CodeBERTScore ( Zhou et al. 2023 ) ; and CodeScore ( Dong et al. 2025 ) , on functionally equivalent code pairs that have varying degrees of surface similarity. In Section 4.3 , we report experimental results demonstrating that all four metrics have moderate to very strong positive correlation with surface similarity. P-values, as low as 10^{-324} , provide statistically significant evidence that these metrics are biased towards how the code looks rather than what it does. We refer to this as the surface bias of a code evaluation metric. We further analyze distinguishability (detailed in Section 3.2.2 ), a meta-metric that evaluates how well a code evaluation metric can distinguish functionally equivalent pairs from the non-equivalent ones. CrystalBLEU has been reported to outperform other baselines in terms of distinguishability ( Eghbali and Pradel 2023 ) . We provide empirical evidence in Section 4.4 that the distinguishability of CEMs considered substantially degrades in the presence of surface bias, with CrystalBLEU itself showing the largest drop of 95.3% . We argue that the performance drop arises because no prior evaluation has explicitly evaluated CEMs in settings where surface similarity and functional behavior move in opposite directions. To address this need, in Section 3.4 we define two contrasting evaluation regions: Similar Form, Different Semantics (SFD), which includes code pairs where the surface similarity largely exceeds functional similarity, and Different Form, Similar Semantics (DFS), which provides for code