CETBench: A Novel Dataset constructed via Transformations over Programs for Benchmarking LLMs for Code-Equivalence Checking

Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg, Parag Singla · 2025

Read CETBench: A Novel Dataset constructed via Transformations over Programs for Benchmarking LLMs for Code-Equivalence Checking by Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg, Parag Singla (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04019.txt.

Introduction

Neeva Oza and co-authors wrote this machine learning paper, published in two thousand twenty-five, on whether language models can tell if two programs do the same thing. Judging code equivalence matters for rewriting and translating software safely.

Their benchmark, CETBench, builds pairs of programs from a shared pool, then applies random chains of small code transformations. Some pairs stay equivalent; others quietly stop being so. Even top models stumble badly after these simple edits, the team finds. Comparing programs across languages makes the drop worse. A light fine-tuning pass on transformed pairs recovers much of the loss. Their generator can vary difficulty, transformation mix, and language pairs. The authors conclude that today's models lean on surface patterns more than true code meaning.

Catalog record

CREATORNeeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg, Parag Singla
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04019

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CETBench: A Novel Dataset constructed via Transformations over Programs for Benchmarking LLMs for Code-Equivalence Checking Neeva Oza, Ishaan Govil, Parul Gupta, Dinesh Khandelwal, Dinesh Garg, Parag Singla Abstract LLMs have been extensively used for the task of automated code generation. In this work, we examine the applicability of LLMs for the related but relatively unexplored task of code-equivalence checking, i.e., given two programs, whether they are functionally equivalent or not. This is an important problem since benchmarking code equivalence can play a critical role in evaluating LLM capabilities for tasks such as code re-writing and code translation. Towards this end, we present CETBench- Code Equivalence with Transformations Benchmark, constructed via a repository of programs, where two programs in the repository may be solving the same or different tasks. Each instance in our dataset is obtained by taking a pair of programs in the repository and applying a random series of pre-defined code transformations, resulting in (non-)equivalent pairs. Our analysis on this dataset reveals a surprising finding that very simple code transformations in the underlying pair of programs can result in a significant drop in performance of SOTA LLMs for the task of code-equivalence checking. To remedy this, we present a simple fine-tuning-based approach to boost LLM performance on the transformed pairs of programs. Our approach for dataset generation is generic, and can be used with repositories with varying program difficulty levels and allows for applying varying numbers as well as kinds of transformations. In our experiments, we perform ablations over the difficulty level of original programs, as well as the kind of transformations used in generating pairs for equivalence checking. Our analysis presents deep insights into the working of LLMs for the task of code-equivalence, and points to the fact that they may still be far from what could be termed as a semantic understanding of the underlying code. 1 Introduction The field of AI-assisted code generation and understanding has witnessed a rapid rise in adoption and capability over the past few years, with models increasingly integrated into real-world software development workflows Jin et al. 2025 . The reasons behind this trend include (i) the emergence of powerful, general-purpose large language models and (ii) the development of standardised benchmarks like HumanEval Chen et al. 2021 and MBPP Austin et al. 2021 , which evaluate text-to-code generation. In this work, our focus is on the related task of checking code-equivalence : given two pieces of code, p1 and p2 , how effective are LLMs in determining whether the two pieces of code are equivalent or not. While there are many definitions of equivalence in the literature, we focus on the most general one, which is functional equivalence , i.e., two programs are said to be equivalent if for any given input, they produce exactly the same output. Doing this task successfully can help with multiple auxiliary tasks, such as verifying the correctness of code rewrites, code translation, etc. While there has been a good amount of work in the classical literature on this task, using AST and program analysis Baxter et al. 1998 ; Gabel et al. 2008 , this is very tedious, does not scale, and often has multiple challenges. Of late, there have been some attempts to examine the capabilities of LLMs for the task of code-equivalence checking, with some of the approaches being parallel to our work Maveli et al. 2025 ; Wei et al. 2025 . We highlight the differences from these works, as well as contrast with other related work in Section 2 . In this work, we provide a general-purpose dataset to evaluate the robustness of LLMs for the undecidable task of code equivalence. The dataset is generated using the following steps: (a) we start with a repository of programs written to solve a variety of programming tasks. (b) We construct a set of pairs of programs by randomly picking programs which (i) solve the same task (either correctly or incorrectly) (ii) solve different tasks. (c) each pair in the set is taken through a series of (randomly picked) pre-defined transformations, such as variable renaming , swap of if-else blocks , etc., to generate the corresponding perturbed pair. We note that some of the transformations are semantic preserving (sp), and others are not (np). These steps result in the dataset of perturbed pairs, and are referred to as CETBench. We perform a detailed analysis of relative performance among different pair categories, before and after introducing the perturbations. Our analysis shows that for a variety of SOTA LLMs, including both open source and closed source, the performance drops significantly on the perturbed dataset. Of specific interest is the drop in performance when the starting pair represents a single program. We present additional insights on LLM performance, we vary the difficulty level MikeMirzayanov 2018 ; Ebtekar 2019 of the original program repository. Additional ablations present the variation in performance across a variety of LLMs, with varying kinds of transformations applied. Finally, we present a simple fine-tuning-based approach to boost LLM performance on the unperturbed as well as perturbed datasets. Best results are obtained when the fine-tuning is done using a combination of pairs from the perturbed as well as unperturbed datasets. Overall, our experiments present deep insights into the working of LLMs for the task of code-equivalence checking for a variety of categories. Further, our findings clearly point to the fact that LLMs are still quite far from semantic understanding of underlying programs, and likely rely on statistical patterns for the task of code equivalence. The contributions of this paper can be summarised as follows: (a) We present a novel dataset of perturbed program pairs generated via a series of code transformations applied randomly to pairs of programs in a repository. (b) We present a detailed analysis demonstrating a significant drop in LLM performance over the perturbed program pairs. We also present additional ablations across varying program difficulty, as well as the type of transformations applied. (c) We present a simple fine-tuning-based approach to boost LLM performance on the perturbed dataset, demonstrating that LLMs do have the capability to work with perturbations but require explicit supervision to effectively deal with this task. 2 Related Work Determining code equivalence—whether two programs yield identical outputs for all possible inputs—is a foundational challenge in computer science, relevant to program verification Chakraborty et al. 2023 , refactoring Liu et al. 2025 , translation Roziere et al. 2020 , and optimization Rosas et al. 2024 . While undecidable in the general case Goldblatt and Jackson 2012 , several heuristic and formal methods have been developed. Traditional Approaches: Static program analysis techniques (e.g., AST comparison Baxter et al. 1998 , data-flow analysis Gabel et al. 2008 ) struggle with semantics-preserving syntactic variations. Formal methods like symbolic execution Kim et al. 2011 and SMT solving Stolee et al. 2014 offer stronger guarantees but often lack scalability. Test-case-based execution is more practical and scalable in real-world settings; however, it has fundamental limitations in establishing true functional equivalence. This limitation arises because test suites inherently cover only a finite subset of potentially infinite program input domains and thus passing these tests merely indicates identical behaviour for those specific inputs rather than universal functional equivalence. Additionally, developing

Read the complete text