SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
Yansong Li 1 , Paula Branco 1 , Alexander M. Hoole 2 , Manish Marwah 2 , Hari Manassery Koduvely 2 , Guy-Vincent Jourdan 1 , Stephan Jou 2 · 2025
Read SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis by Yansong Li 1 , Paula Branco 1 , Alexander M. Hoole 2 , Manish Marwah 2 , Hari Manassery Koduvely 2 , Guy-Vincent Jourdan 1 , Stephan Jou 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20630.txt.
Introduction
Yansong Li and colleagues set out a benchmark to test if large language models can be trusted to study flaws in source code. Models more and more find and fix such flaws. But tests rarely probe the thought behind their answers. Trust calls for two kinds of thought: thought about code structure, and thought about what code means.
Their benchmark, SV-TrustEval-C, targets C code on those two axes. Structure thought is tested as data flow and control flow grow more complex. Meaning thought is tested by changing code in form and in sense, then checking if the model stays sound. Results show current models fall far short on complex code links. Their work rests more on pattern matching than on robust thought. The benchmark gives builders a sharper tool.
Catalog record
| CREATOR | Yansong Li 1 , Paula Branco 1 , Alexander M. Hoole 2 , Manish Marwah 2 , Hari Manassery Koduvely 2 , Guy-Vincent Jourdan 1 , Stephan Jou 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20630
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis Yansong Li 1 , Paula Branco 1 , Alexander M. Hoole 2 , Manish Marwah 2 , Hari Manassery Koduvely 2 , Guy-Vincent Jourdan 1 , Stephan Jou 2 Abstract As Large Language Models (LLMs) evolve in understanding and generating code, accurately evaluating their reliability in analyzing source code vulnerabilities becomes increasingly vital. While studies have examined LLM capabilities in tasks like vulnerability detection and repair, they often overlook the importance of both structure and semantic reasoning crucial for trustworthy vulnerability analysis. To address this gap, we introduce SV-TrustEval-C , a benchmark designed to evaluate LLMs’ abilities for vulnerability analysis of code written in the C programming language through two key dimensions: structure reasoning—assessing how models identify relationships between code elements under varying data and control flow complexities; and semantic reasoning—examining their logical consistency in scenarios where code is structurally and semantically perturbed. Our results show that current LLMs are far from satisfactory in understanding complex code relationships and that their vulnerability analyses rely more on pattern matching than on robust logical reasoning. These findings underscore the effectiveness of the SV-TrustEval-C benchmark and highlight critical areas for enhancing the reasoning capabilities and trustworthiness of LLMs in real-world vulnerability analysis tasks. Our initial benchmark dataset is available at https://huggingface.co/datasets/LLMs4CodeSecurity/SV-TrustEval-C-1.0 I Introduction With the continued development of Large Language Models (LLMs) [ 1 , 2 , 3 ] , specialized versions tailored for the code domain [ 4 , 5 , 6 ] , have demonstrated promising capabilities in code understanding and generation. Notable applications such as Amazon CodeWhisperer and GitHub Copilot have significantly boosted developer productivity and been widely adopted by millions of developers [ 7 ] . Building on these advances, numerous studies have broadened the applications of LLMs to include cybersecurity tasks, particularly in code vulnerability analysis [ 8 , 9 , 10 ] . Unlike traditional tools that depend on post-hoc verification through static [ 11 ] or dynamic analysis [ 12 ] , LLMs offer real-time security feedback during the coding process [ 13 ] and can generate secure code while following detailed instructions [ 14 ] , significantly reducing the risk of software vulnerabilities. Despite their potential, studies have shown that LLMs do not guarantee code security and are prone to generating syntactically correct but semantically insecure code [ 15 , 16 , 17 ] . This raises concerns about their utility in code vulnerability analysis and underscores the urgent need to assess their capabilities in the security domain [ 18 , 19 ] . However, existing benchmarks for code vulnerability, such as CVEFixes [ 20 ] , BigVul [ 21 ] , and DiverseVul [ 22 ] , primarily focus on straightforward tasks like identifying or repairing publicly known vulnerable code snippets within predefined scenarios. These benchmarks fall short as they fail to discern whether success in resolving vulnerabilities stems from leveraging pre-trained parameterized knowledge [ 23 , 24 ] , or from a logical reasoning between code and vulnerabilities. This distinction is vital for verifying the reliability of LLMs, as reliance exclusively on pre-trained patterns often leads to poor generalization [ 25 ] and inconsistent interoperability [ 26 ] , especially when addressing semantic and structural variations in code within real-world scenarios. To overcome these limitations, we formally define the core capabilities required for vulnerability analysis and introduce SV-TrustEval-C , a new question-answering benchmark designed for the trustworthy evaluation of LLMs in source code vulnerability analysis. This benchmark involves: Structure Reasoning: Unlike unstructured natural language, code inherently possesses rigorous structural information. For natural language-driven LLMs, the ability to comprehend this code structure is essential for effective code understanding and reasoning [ 27 , 28 ] . In vulnerability analysis, understanding the relationships between code elements is critical for assessing the scope and impact of vulnerabilities and for implementing accurate repair strategies. This benchmark evaluates the ability of LLMs to identify and predict the effects of modifications within code elements on each other. Semantic Reasoning: Leveraging domain knowledge to adapt to changes in code is crucial for maintaining robustness in dynamic environments. This benchmark component evaluates LLMs through scenarios that simulate real-world conditions, including: Counterfactual scenarios , where altered code semantics challenge LLMs to apply logical reasoning beyond learned patterns; Goal-driven scenarios , where LLMs are tasked with completing code while ensuring functionality and security, testing their ability to handle complex modifications without introducing vulnerabilities; and Predictive scenarios , where LLMs must accurately identify and differentiate between various types of vulnerabilities, including scenarios where vulnerabilities will not be triggered at runtime, thereby assessing their application of security concepts. To develop SV-TrustEval-C , we created a Structure-Oriented Variants Generator capable of extracting structural information, perturbing code semantics based on the provided base code and classifications ( Safe , Unsafe code) 1 1 1 Safe and Unsafe : defined by whether the code triggers a vulnerability. , and increasing code complexity in alignment with data flow and control flow graphs [ 29 ] . We conducted extensive experiments using the SV-TrustEval-C benchmark with zero-shot and in-context learning inference to evaluate eleven popular LLMs across various parameter scales. The results reveal that LLMs struggle to recognize relationships between code elements and predominantly rely on pattern matching rather than logical reasoning in vulnerability analysis scenarios. This highlights the need for specialized improvements in LLMs to enhance their practical utility in real-world code vulnerability analysis. Our contributions are the following: 1. Reasoning-based Benchmark for Vulnerability Analysis: We present SV-TrustEval-C , the first benchmark designed to assess LLMs’ ability to analyze source code vulnerabilities through logical reasoning and structural understanding, moving beyond mere pattern recognition. 2. Structure-Oriented Variants Generator: We created a generator that systematically extracts structural information, alters code semantics, and increases complexity based on data and control flow graphs using Safe and Unsafe code pairs. 3. Identifying Gaps in LLM Capabilities: Evaluating eleven LLMs revealed their reliance on pattern matching over logical reasoning in vulnerability analysis, highlighting the need for enhancing their reasoning capabilities in security applications. TABLE I : Comparison of recent code vulnerability evaluation benchmarks against our proposed SV-TrustEval-C . “CWE Scope” indicates the number of CWEs in a benchmark; “Num. of Func.” shows the total functions evaluated. Icons indicate dataset origin— for manually labeled real‑world data, for automatically labeled real‑world data, and for synthetic data. Support levels are shown as ✓ (full), $\checkmark$⃝ (partial), and ✘ (none). Key features of our benchmark—CWE coverage, function count, and reasoning capability—are highlighted relative to SecLLMHolmes . Evaluation tasks: Identification (Eq. 1 and Eq. 2 ) measures vulnerability detection and localization; Repair (Eq. 3 ) assesses conversion of vulnerable code into secure versions; Generation (Eq. 4 ) evaluates producing safe code without introducing new vulnerabilities; QA (Eq. 5 ) tests domain knowledge via targeted question‑answering; and Reasoning (Eq. 6 ) evaluates the underlying reasoning process. Benchmark CWE Scope Num. of Func. Source Identification Repair Generation QA Reasoning SVEN [ 14 ] 9 1,606 ✘ ✘ ✓ ✘ ✘ Devign [ 30 ] N/A 26,037 ✓ ✓ ✘ ✘ ✘ VulPatchPairs [ 31 ] N/A 26,000 ✓ ✓ ✘ ✘ ✘ BigVul [ 21 ] 91