Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models

Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Yi Li · 2025

Read Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models by Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Yi Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-11686.txt.

Introduction

Jian Wang and 4 co-authors test whether execution traces actually help code language models reason in a 2025 study. Such models impress at writing code, yet they struggle to reason about what programs do at runtime, and semantic information like traces is represented inconsistently across existing methods.

The authors introduce a generic framework for adding trace-based semantic information to task prompts in a consistent way. They then study its usefulness for supervised fine-tuning and for inference-time scaling. Surprisingly, results disagreed with previous work. Semantic information showed limited usefulness in both settings studied. That negative finding matters because it redirects effort: simply feeding models execution traces does not reliably fix their weak runtime reasoning.

Catalog record

CREATORJian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Yi Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-11686

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, Yi Li Abstract Code Large Language Models (Code LLMs) have opened a new era in programming with their impressive capabilities. However, recent research has revealed critical limitations in their ability to reason about runtime behavior and understand the actual functionality of programs, which poses significant challenges for their post-training and practical deployment. Specifically, Code LLMs encounter two principal issues: (1) a lack of proficiency in reasoning about program execution behavior, as they struggle to interpret what programs actually do during runtime, and (2) the inconsistent and fragmented representation of semantic information, such as execution traces, across existing methods, which hinders their ability to generalize and reason effectively. These challenges underscore the necessity for more systematic approaches to enhance the reasoning capabilities of Code LLMs. To address these issues, we introduce a generic framework to support integrating semantic information (e.g., execution trace) to code task-relevant prompts, and conduct a comprehensive study to explore the role of semantic information in enhancing the reasoning ability of Code LLMs accordingly. Specifically, we focus on investigating the usefulness of trace-based semantic information in boosting supervised fine-tuning (SFT) and post-phase inference of Code LLMs. The experimental results surprisingly disagree with previous works and demonstrate that semantic information has limited usefulness for SFT and test time scaling of Code LLM. 1 Introduction Code large language models (Code LLMs) have emerged as prominent programming assistants, demonstrating remarkable performance across various coding tasks, including program repair Xia et al. (2022) , code generation Liu et al. (2023b) , and code summarization Jain et al. (2020) . Recently, several Code LLMs have been introduced, each characterized by distinct training schemes. For instance, Llama3.1 Dubey et al. (2024) ; Roziere et al. (2023) , is fine-tuned with code infilling tasks and long code input contexts, complemented by an instruction fine-tuning process. Similarly, DeepSeek-Coder Guo et al. (2024) is trained on more than 2 trillion tokens using a fill-in-the-blank task to enhance its code generation capabilities. These models focus on learning contextual information from code and docstrings, advancing their general understanding of code Chen et al. (2024) ; Ni et al. (2024) ; Ding et al. (2024) . However, these approaches predominantly capture the static dimensions of code (e.g., tokens and context), while neglecting the dynamic semantics crucial for a comprehensive understanding of code. Recent studies have highlighted this limitation, revealing that even state-of-the-art models like GPT-4 struggle to reason about runtime behaviors of code Chen et al. (2024) . Understanding code semantics and accurately predicting runtime behavior is critical, particularly for practical coding applications that require semantic understanding Ni et al. (2024) ; Ding et al. (2024) . This underscores the urgent need to investigate methodologies that enhance the reasoning capabilities of Code LLMs for better semantic understanding, such as coverage prediction Chen et al. (2024) and output prediction Chen et al. (2024) ; Liu et al. (2023a) abilities. To address these challenges, two primary methods have been proposed to bolster the reasoning capabilities of Code LLMs in code generation tasks: 1) inference with semantic information – integrating feedback, such as error information to the inputs (or outputs) at inference time, and 2) fine-tuning with semantic information – directly enhancing the models’ intrinsic reasoning by integrating semantically enriched fine-tuning data, thereby enabling improved predictions Ni et al. (2024) ; Ding et al. (2024) . In this work, we consider both approaches and explore whether and how incorporating code semantic information can enhance the performance of Code LLMs. Specifically, we investigate evaluate their effectiveness across various code-related tasks. The core challenge lies in identifying and collecting appropriate semantic data for training or inference to improve the semantic comprehension of Code LLMs. Recent efforts Ding et al. (2024) ; Ni et al. (2024) have begun to address this by fine-tuning models using dynamic data, such as execution behaviors, to enhance semantic understanding. While these approaches have shown promise, they employ diverse semantic representations, such as natural language descriptions of programs or execution traces. There remains a lack of 1) a unified study and tool that supports all semantic representations, and 2) systematic understanding regarding how different training data compositions, particularly in terms of semantic representations, impact code reasoning and generation capabilities. To this end, we propose and implement a generic framework that facilitates the generation of multiple types of semantic representations, supporting post-training, one-time inference, and the scaling of test-time computation during inference. Based on this framework, we conduct a systematic study to explore the efficacy of semantic information in boosting code generation. Specifically, we integrate different semantic representations (i.e., different code execution traces) into the input data (prompt) during both SFT time and inference time to assess their impact. Additionally, we examine the influence of different training strategies (e.g., parameter-efficient fine-tuning) on the effectiveness of semantic information. Different from previous research findings Ni et al. (2024) ; Ding et al. (2024) , our experimental results demonstrate that integrating existing semantic information into the input prompt has limited benefits to the performance of tuned Code LLMs. To summarize, the main contributions of this paper are: • We introduce the first generic framework that supports different types of code semantic representations. Based on this framework, we construct and open-source a high-fidelity dataset featuring diverse execution behavior representations, including bug-patch function pairs, unit tests, and multiple semantic layers. The dataset and all related implementations are publicly available on our website 1 1 1 https://github.com/tracewise-probing/tracewise_probing . • We conduct a comprehensive study to explore the effectiveness of semantic information in enhancing both SFT and inference of code LLMs. • We summarize multiple findings, such as semantic information integrated in the input does not positively contribute to the inference. 2 Problem Statement We address the problem of enhancing the code generation capabilities of Code LLMs by integrating code semantic information into the input prompts. Current Code LLMs primarily rely on static text data, which often fails to capture the nuanced semantics crucial for thorough code understanding Wei et al. (2023) ; Wei et al. (2024) ; Abdin et al. (2024) . Inspired by the practices of human developers, who iteratively refine code through reasoning and semantic assessment rather than relying solely on dynamic testing or runtime feedback, we note that such reasoning and refinement processes are largely absent in existing models Ni et al. (2024) ; Ding et al. (2024) . There are two main sub-problems we are interested in for Code LLMs, 1) fine-tuning with semantic information, and 2) inference with semantic information. (1) Fine-tuning with semantic information. Our Code LLM fine-tuning paradigm focuses on a repair-based fine-tuning framework that leverages a carefully curated dataset \mathcal{D}\;=\;\bigl\{\,(x,y)\;\big|\;x=(b,r),\;y=a\bigr\} where (x, y) is the input-output pair for model fine-tuning, b denotes the buggy code fragment, r is execution-trace rationale, and a represents the patched code. We selected the code repair task because models often fail to produce correct code in a single attempt, requiring the refinement of the output until it is correct. A central question is how to encode the reasoning signals r so that the Code LLM can learn better code generation capability.

Read the complete text