Representation Matters: An Empirical Study of Program Representations for LLM Vulnerability Reasoning
Andrew Stoltman, Johnathan Tang, Haipeng Cai · 2026
Read Representation Matters: An Empirical Study of Program Representations for LLM Vulnerability Reasoning by Andrew Stoltman, Johnathan Tang, Haipeng Cai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25356.txt.
Introduction
Feeding raw source code to a language model hunting for security flaws feels natural, but more context may not mean better reasoning. Andrew Stoltman, Johnathan Tang, and Haipeng Cai test that assumption in 2026 with a benchmark covering one hundred seven real world C and C plus plus vulnerability cases. Each case was converted into several forms: raw code, syntax trees, control flow graphs, dependence graphs, and combinations. Ten variants faced the same fixed reasoning prompt and output format.
Combining a syntax tree with a dependence graph reached accuracy in the low eighties, against the mid fifties for raw source alone. Graph only prompts beat both source only and mixed prompts while using far fewer tokens. Adding raw code to compact graph evidence often hurt, a dilution effect the authors highlight for security focused model design.
Catalog record
| CREATOR | Andrew Stoltman, Johnathan Tang, Haipeng Cai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 90 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Representation Matters: An Empirical Study of Program Representations for LLM Vulnerability Reasoning Andrew Stoltman, Johnathan Tang, Haipeng Cai Abstract Large Language Models (LLMs) are increasingly adopted for automated vulnerability detection, but the optimal way to represent program structure and semantics for LLM-based vulnerability reasoning remains unclear. Most existing prompting-based approaches expose the model directly to raw source code, implicitly assuming that more source-level context provides better evidence. This paper challenges that assumption through a systematic empirical benchmark of raw source code and various static-analysis-based code representations for LLM vulnerability analysis. We present RepBench , a unified benchmark pipeline that converts real-world C/C++ vulnerability testcases into multiple program representations—raw source code, Abstract Syntax Trees (ASTs), Control-Flow Graphs (CFGs), Program Dependence Graphs (PDGs), and representation combinations thereof—along with an auxiliary track of enriched Program Dependence Graphs (ePDGs). Using a curated PrimeVul-derived corpus of 107 standard Joern-based testcases across five CWE categories, we evaluate ten representation variants under a fixed Chain-of-Thought (CoT) prompting and structured-output protocol, with an additional 19 auxiliary ePDG cases generated through an external VulChecker/Hector pipeline. Our results show that representation choice has a substantial effect on LLM vulnerability reasoning. The strongest variant, AST+PDG, achieves 83.2% curated accuracy compared with 53.5% for the raw-source baseline—a 29.7 percentage point improvement. At the prompt-family level, graph-only prompts outperform both source-only and source-plus-graph prompts while requiring substantially lower prompt overhead. This reveals a counter-intuitive context dilution effect: adding raw source code to compact structural graph evidence often degrades reasoning performance rather than improving it. Our findings suggest that carefully selected structural representations offer a stronger accuracy–prompt-overhead tradeoff than simply exposing LLMs to more raw input, and that static analysis can serve as an effective prompt-construction layer for security-focused LLM reasoning. 1 Introduction LLMs have rapidly become a major component of modern software engineering workflows, including code generation, debugging, test generation, program repair, and vulnerability analysis. In software security, LLMs are increasingly used to inspect code, explain suspicious behaviors, identify vulnerable statements, and reason about whether a function is vulnerable or safe [ 28 , 35 ] . Despite this growing adoption, a foundational design question remains underexplored: what form of program representation should be given to an LLM when asking it to reason about security-relevant code behavior? Most existing LLM-based vulnerability analysis pipelines use raw source code as the primary input representation [ 28 ] . This is natural because LLMs are trained on large corpora of source code and can interpret syntax, idioms, and API usage directly. However, vulnerability reasoning often depends on program relations that are not explicit in surface syntax. Identifying an integer overflow may require understanding whether arithmetic operations are guarded by appropriate bounds checks [ 15 ] . Detecting a double free or use-after-free may require reasoning about allocation, deallocation, pointer aliases, and subsequent reuse [ 15 ] . These properties are precisely what software-analysis representations—e.g., control-flow graphs and program dependence graphs—have been designed to expose. This paper studies the hypothesis that structured program representations can help LLMs reason about vulnerability semantics more effectively than raw source code, because they expose security-relevant structural evidence in a compact and focused form . We evaluate this hypothesis through a controlled representation-ablation benchmark. The vulnerability detection task is held fixed: for each testcase, the LLM must classify a target function as VULNERABLE or SAFE for a specified CWE. The only experimental factor varied is the input representation provided to the model. Accordingly, we build RepBench , a benchmark framework that transforms real-world vulnerability testcases into multiple textualized program representations and evaluates them under a fixed LLM vulnerability-detection task. The current study draws from PrimeVul-derived C/C++ examples [ 2 ] and focuses on five CWE categories: CWE-122, CWE-190, CWE-191, CWE-415, and CWE-416. The main evaluation track uses Joern [ 32 ] to extract AST, CFG, and PDG views; an auxiliary track evaluates enriched PDGs (ePDGs) generated through the VulChecker/Hector pipeline [ 14 ] . Across all variants, the task remains the same: given a target function and a specified CWE, the model must classify the function as Vulnerable or Safe . Only the program representation given to the model is varied. Specifically, our study is organized around four main research questions: • RQ1: How effectively do standalone structural graph representations support LLM-based vulnerability reasoning compared to raw source code? • RQ2: Which program representations and representation combinations are most effective for LLM vulnerability reasoning? • RQ3: What tradeoff exists between representation effectiveness and prompt overhead? • RQ4: How do representation effects vary across vulnerability categories, ground-truth labels, and auxiliary ePDG inputs? Our evaluation reveals a strong and counter-intuitive representation effect on LLM performance in vulnerability reasoning, including four main findings aligned with these questions. For RQ1 , structural graph representations substantially outperform raw source-code prompting . The raw-source baseline achieves only 53.5% curated accuracy, while every evaluated graph-based variant improves over it. At the prompt-family level, graph-only prompts reach 74.4% accuracy. For RQ2 , representation choice matters not only at the family level but also among individual structural views, and combining complementary graph views is especially effective . The strongest variant is AST+PDG, which achieves 83.2% curated accuracy, a 29.7 percentage-point improvement over raw source. AST+CFG is the second strongest variant, achieving 80.4% curated accuracy. These results suggest that LLM vulnerability reasoning benefits from combining local syntactic anchors with either dependence evidence or control-flow evidence. However, source-plus-graph variants do not consistently improve over, and often underperform, graph-only variants, indicating that adding raw source code can introduce distracting context—not useful complementary evidence. For RQ3 , graph-only representations provide the best effectiveness–overhead tradeoff. They achieve the highest family-level accuracy while using substantially fewer prompt characters and input tokens than source-plus-graph prompts. In particular, source-plus-graph prompts require 2.4 \times as many characters and 2.2 \times as many input tokens as graph-only prompts, yet achieve lower accuracy. This result reveals what we call the context dilution effect: adding more input context can reduce the salience of vulnerability-relevant evidence, causing the model to attend to irrelevant or misleading content despite receiving more total information. Our case studies further show that this effect is not merely a consequence of prompt truncation; source-augmented prompts can fail even when they remain well within the prompt budget. For RQ4 , representation effects are broadly visible across the adequately supported CWE categories, but they are not uniform. AST+PDG improves over raw source on CWE-190, CWE-415, and CWE-416, suggesting that the aggregate graph advantage is not solely an artifact of the CWE-190-heavy corpus . At the same time, source-plus-graph prompts perform competitively on some memory-lifecycle cases, indicating that raw source context may still help for certain vulnerability patterns. Label-wise analysis further shows that the model is more reliable on SAFE cases than on VULNERABLE cases. Finally, the auxiliary ePDG track shows that vulnerability-oriented dependence metadata can support correct reasoning in some cases , but its current effectiveness is limited by LLVM-level readability and artifact-generation feasibility. Together, these findings have implications beyond this specific benchmark. LLM-based software analysis systems that rely solely on raw source prompting may fail to exploit decades of progress in static analysis. Conversely, representation-aware systems can use program analysis as a semantic compression layer , presenting models with compact structural