Can Large Language Models Solve Path Constraints in Symbolic Execution? DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generate the Correct
Wenhan Wang, Kaibo Liu, Zeyu Sun, An Ran Chen, Ge Li, Gang Huang, Lei Ma · 2025
Read Can Large Language Models Solve Path Constraints in Symbolic Execution? DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generate the Correct by Wenhan Wang, Kaibo Liu, Zeyu Sun, An Ran Chen, Ge Li, Gang Huang, Lei Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-18288.txt.
Introduction
Wenhan Wang and six co-authors ask if large language models can reason about the paths a Python program may take. Path reasoning underpins test design and bug hunts that avoid running the code. Classic tools use logic solvers, but they stall on rich data forms and calls to outside code, and Python's loose syntax leaves few such tools in wide use.
The study sets models two kinds of path tasks, to make paths and to judge them. It weighs how well the models trace branch choices through real code. The results map where model reasoning holds and where it breaks. That map helps judge if model help can fill the tool gap for a widely used language.
Catalog record
| CREATOR | Wenhan Wang, Kaibo Liu, Zeyu Sun, An Ran Chen, Ge Li, Gang Huang, Lei Ma |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-18288
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Large Language Models Solve Path Constraints in Symbolic Execution? DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper Wenhan Wang, Kaibo Liu, Zeyu Sun, An Ran Chen, Ge Li, Gang Huang, Lei Ma Abstract . Symbolic execution is an important software analysis technique which benefits downstream tasks such as software testing and debugging. However, several limitations hinder symbolic execution from application on real-world software. One of the limitations is the inability to solve diverse execution path constraints: traditional symbolic execution based on SMT solvers is difficult to handle execution paths with complex data structures or external API calls. In this paper, we focus on investigating the possibility of adopting large language models (LLM) for path constraint solving instead of traditional solver-based techniques in symbolic execution. We conduct an empirical study to evaluate the ability of LLMs in two types of path constraint solving: generating test inputs to facilitate an execution path, and determining whether a given execution path can be satisfied without triggering any bugs. We build new evaluation pipelines and benchmarks for two tasks: test case generation and path classification, which include data sources from both competition-level programs and real-world repositories. Our experiment results show that state-of-the-art LLMs are able to solve path constraints in both generation and classification tasks, with 60% of generated test cases that accurately cover the given execution path. Moreover, LLMs are capable of improving test coverage by covering execution paths in real-world repositories where traditional symbolic execution tools cannot be applied. These findings highlight the possibility of extending symbolic execution techniques with LLMs in the future to improve the ability and generalizability of symbolic execution. 1. Introduction Symbolic execution ( King 1976 ; Baldoni et al. 2018 ) is an important program analysis technique that supports a wide range of software engineering applications, such as testing ( Cadar et al. 2008 ) , bug finding ( Ramos and Engler 2015 ) , and verification ( Coen-Porisini et al. 2001 ) . The key idea of symbolic execution is to execute the program with symbolic inputs instead of concrete values, and simultaneously explore different execution paths. During this process, symbolic execution first traverses the program to collect the path condition constraints and subsequently employs an SMT solver ( De Moura and Bjørner 2008 ) (e.g., Z3 or CVC5) to determine their satisfiability and predict possible values for symbolic variables. Although symbolic execution has become a mature technique for software engineering, several critical challenges still hinder its broader adoption in general-purpose program analysis. Among these challenges, one of the most fundamental lies in solving complex path constraints. First, for some program languages (e.g., Python with its flexible grammar), translating path constraints from source code into SMT solver expressions remains non-trivial ( Wang et al. 2024b ) . Moreover, the complexity of some constraints often exceeds the solving capabilities of current SMT solvers ( Li et al. 2025a ; Wu et al. 2025 ) . For example, traditional symbolic execution still struggles to handle external API calls and complex data structures. Recently, large language models (LLMs) have demonstrated remarkable capabilities in both code generation and code understanding ( Jimenez et al. 2024 ; Gu et al. 2024 ) , achieving performance comparable to, or even surpassing, that of human experts across a wide range of tasks. More significantly, LLMs have begun to reshape areas of software engineering that were once dominated by symbolic execution, including automated testing ( Wang et al. 2024a ; Wang et al. 2025 ; Jain et al. 2025 ) and bug detection ( Li et al. 2024 ) . These successes suggest that LLMs possess a latent understanding of program semantics and execution paths, suggesting the potential to complement or enhance symbolic execution techniques. The recent emergence of large reasoning models (LRM) ( Li et al. 2025b ) further strengthens this potential. By integrating reinforcement learning, LRMs significantly strengthen the reasoning abilities of LLMs, enabling them to generate extended chains of thought (CoT) ( Wei et al. 2022 ) before producing final answers. These models excel at complex logical reasoning tasks—such as mathematical problem solving and theorem proving—that form the very foundation of symbolic execution. Since symbolic execution intrinsically relies on constraint solving and systematic path exploration through logical reasoning, it appears to be a natural beneficiary of LLM-driven enhancement. This convergence of needs and capabilities raises a pivotal question: Can LLMs empower symbolic execution to advance software engineering tasks? In this paper, we conduct a systematic empirical study on the efficacy of LLMs in solving path constraints for symbolic execution, and analyze the different behaviors of non-reasoning LLMs and LRMs. We focus on evaluating LLMs on two different tasks, which include both generation-style and classification-style. The first one is generating test cases to satisfy a given execution path, which is a typical application of symbolic execution. The second task is execution path classification: determine whether a path is valid or if it can trigger a bug. To evaluate these tasks, we construct benchmarks derived from both programming competition problems—characterized by intricate control flows—and real-world software repositories, which often involve the use of third-party libraries. We aim to address the following research questions: • RQ1: Can LLMs generate test cases for path constraints? We investigate the power of LLMs in test case generation, one of the most commonly used generation tasks for symbolic execution. For this task, we evaluate LLMs by prompting them to generate test cases for Python programs to cover given execution paths. Although prior work ( Ryan et al. 2024 ) has attempted to enhance LLM-based test case generation with execution path information, it lacks support for paths with complex control flow structures, such as loops, and it remains unexplored whether the generated test case can precisely cover the given path. • RQ2: Can LLMs categorize path constraints by their feasibility and bug-triggering behaviors? Symbolic execution is also widely used in classification-style tasks, such as classifying path feasibility for bug detection. For this RQ, we extract both feasible and infeasible execution paths by designing a path traversal algorithm on control flow graphs (CFGs). Based on these extracted paths, we then evaluate whether LLMs can accurately discriminate among valid, infeasible, and bug-triggering execution paths. • RQ3: Can LLMs handle path constraint solving in real-world software repositories? In the previous two RQs, we mainly focused on path constraint solving in standalone programs without external dependencies, which does not reflect real-world software development practices. Among the key challenges of symbolic execution, external API calls represent a major obstacle that hinders the applicability of traditional symbolic execution to practical software systems. For this RQ, we investigate whether LLM-based path constraint solving can help software testing on real-world software repositories, i.e., if prompting LLMs to cover designated execution paths can improve test coverage. Our study shows several findings