LLMs in Code Vulnerability Analysis: A Proof of Concept Conference: Proceedings of the Fourth International Workshop on Software Vulnerability Management (SVM 2026), Rio De Janeiro,Brazil; April 12–18, 2026; Porto de Galinhas, Brazil CCS: Software and its engineering Software testing and debugging

Shaznin Sultana, Sadia Afreen, Nasir U. Eisty · 2026

Read LLMs in Code Vulnerability Analysis: A Proof of Concept Conference: Proceedings of the Fourth International Workshop on Software Vulnerability Management (SVM 2026), Rio De Janeiro,Brazil; April 12–18, 2026; Porto de Galinhas, Brazil CCS: Software and its engineering Software testing and debugging by Shaznin Sultana, Sadia Afreen, Nasir U. Eisty (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08691.txt.

Introduction

Shaznin Sultana, Sadia Afreen, and Nasir Eisty test language models for code vulnerability analysis. Their 2026 arXiv proof of concept notes that traditional security analysis struggles with modern codebase scale and complexity. Intelligent automation is needed to detect, assess, and remediate flaws efficiently. The authors explore code-specific and general-purpose language models for these tasks. Tasks include identifying vulnerabilities, predicting severity and access complexity, and generating fixes. The study gauges whether such models are ready to assist security teams in workshops and practice.

Catalog record

CREATORShaznin Sultana, Sadia Afreen, Nasir U. Eisty
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08691

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLMs in Code Vulnerability Analysis: A Proof of Concept Conference: Proceedings of the Fourth International Workshop on Software Vulnerability Management (SVM 2026), Rio De Janeiro,Brazil; April 12–18, 2026; Porto de Galinhas, Brazil CCS: Software and its engineering Software testing and debugging Shaznin Sultana, Sadia Afreen, Nasir U. Eisty Abstract . Context: Traditional software security analysis methods struggle to keep pace with the scale and complexity of modern codebases, requiring intelligent automation to detect, assess, and remediate vulnerabilities more efficiently and accurately. Objective : This paper explores the incorporation of code-specific and general-purpose Large Language Models (LLMs) to automate critical software security tasks, such as identifying vulnerabilities, predicting the severity and access complexity rate, and generating fixes as a proof of concept. Method : We evaluate five pairs of the most recent LLMs, which contain both code-based and general-purpose open-source models, on two recognized C/C++ vulnerability datasets, namely Big-Vul and Vul-Repair. Additionally, we show comparisons between fine-tuning and prompt-based approaches. Results : The results show that fine - tuning uniformly outperforms both zero-shot and few - shot approaches across all tasks and models. Notably, code - specialized models excel in zero - shot and few - shot settings on complex tasks, but general - purpose models remain nearly as effective. Discrepancies among CodeBLEU, CodeBERTScore, BLEU, and ChrF highlight the inadequacy of the current metrics for measuring repair quality. Conclusions : Through this study, we contribute to the resilience of the current software security community by investigating the potential of advanced LLMs. 1. Introduction Traditional vulnerability assessment tools, such as static analyzers, have limitations in coverage, programming language support, and vulnerability identification accuracy ( ghaleb_how_2020 ) . Some major problems developers encounter include the likelihood of bugs, the need for frequent configuration, and the difficulty of using tools ( beller_analyzing_2016 ; vassallo_how_2020 ) . It is challenging to detect complicated vulnerabilities manually or using traditional tools in large, correlated code bases. Modern software security requires intelligence that adapts rather than static tools and manual code reviews. Using automated vulnerability detection systems based on machine learning and deep learning to monitor, evaluate, and respond to potential threats is much more efficient than human methods ( vuldetect_syslit ; steenhoek_empirical_2023 ; adriana2024toward ) . With the tremendous development of LLMs ( LLM_godfatherpaper ) in large-scale text and code analysis, they are being investigated beyond general natural language processing (NLP) tasks in diverse areas ( survey_llm ; LLM_taxonomy ) . LLMs are making great progress in software engineering ( llm_in_SE ) . It has led to further exploration about whether LLMs can complement critical fields like vulnerability analysis ( repairComprehensice ) or whether human experts are still mandatory to some extent ( LLM_notyet ) . However, LLMs’ practical value, adaptability, and cost efficiency require a more comprehensive study beyond detecting vulnerabilities, such as analyzing their severity and impact. As a proof of concept for the SVM workflow, this study assesses open-source LLMs to streamline software security. While existing studies largely focus on vulnerability detection and repair, we introduce a more granular evaluation that benchmarks five pairs of code-specific versus general-purpose LLMs across four-stage pipeline. Starting with detection to risk assessment, where severity and access complexity predictions inform the prioritization of critical vulnerabilities, facilitating the repair generation. We focus on clarifying the benefits and limits of LLMs by comparing their performance to recognized metrics. In this process, we utilize two widely used large datasets, Big-vul ( bigvul ) and Vul-Repair ( Chen_2023vulrepair ) , on C/C++ programs to make the comparison. For the LLM selection, we chose five pairs of code-based and general open-source models. Through our experiments, we address the following research questions to uncover significant insights into the effectiveness of LLMs in contributing to code vulnerability analysis. RQ1: : How does the effectiveness of fine-tuning compare to prompt-based approaches like zero-shot and few-shot learning? RQ2: : Do code-specific models consistently outperform general-purpose models across different approaches, or is it the other way around? RQ3: : Which pair of models shows superior performance relative to other pairs? RQ4: : Are existing code similarity metrics adequate for evaluating the capabilities of LLMs in code fix generation tasks? The contributions of our study are as follows: • We conduct a thorough evaluation to determine if code-specialized models offer significant advantages over general-purpose models in vulnerability analysis. • We compare prompt-based methods with fine-tuned approaches to assess whether the efficiency of prompting compensates for the higher resource demands of fine-tuning in handling complex vulnerabilities. • We evaluate open-source LLMs to explore their potential as viable alternatives to proprietary models. • We have made our code and dataset available at - https://figshare.com/s/a06ec09cd1bd98e6dd45 2. Related Works Recent advances in LLMs have inspired research into software vulnerability detection ( LLMagent_2025 ) . Early efforts modified BERT or combined sequence–graph embeddings ( hou_large_2023 ) , while Zhou et al. ( zhou2024emergingresults ) used GPT-3.5/4 with in-context learning to identify bugs. Beyond detection ( vuldetectbench2024 ; enhancingreverseengineering2024 ; closingthegap2025 ; islam2024enhancingsourcecodesecurity ; Multitask2024 ) , studies examined LLMs’ abilities in vulnerability description ( enhancingreverseengineering2024 ; Multitask2024 ; islam2024enhancingsourcecodesecurity ) , localization ( closingthegap2025 ; Multitask2024 ) , repair ( islam2024enhancingsourcecodesecurity ; chatgpt2023 ) , classification ( vuldetectbench2024 ; enhancingreverseengineering2024 ; closingthegap2025 ; chatgpt2023 ) , and root-cause analysis ( vuldetectbench2024 ; enhancingreverseengineering2024 ) . Most studies highlight fine-tuning as effective. Liu et al. ( vuldetectbench2024 ) showed that larger models perform better on simple tasks but struggle with root-cause analysis. Islam et al. ( islam2024enhancingsourcecodesecurity ) improved repair by 12% using SecRepair and reinforcement learning. Zhou et al. ( review2024 ) offered a roadmap for adapting and deploying LLMs in workflows. Prompt design also matters. Liu et al. ( liu2024prompting ) found that combining code details with CoT prompts boosts GPT-3.5 turbo. Mathews et al. ( mathews_llbezpeky_2024 ) used retrieval-augmented prompts, emphasizing the use of structured pipelines. Manuel et al. ( enhancingreverseengineering2024 ) fine-tuned on decompiled code but noted dataset bias. LLMs also support repair generation ( wu2023neuralnetworks ; wang2024aigeneratedcodereallysafe ; repairComprehensice ; david2024enhancedrepair ; zhou_large_2024 ) . Huang et al. ( repairempirical2023 ) improved accuracy by marking bug locations, though models falter on long sequences. Zhou et al. ( zhou2024multillmcollaborationdatacentric ) localized bugs with <start><end> tokens. Boyang et al. ( yang2024multiobjectivefinetuningenhancedprogram ) introduced MOREPAIR, optimizing repair and explanation jointly. Hybrid approaches like IRIS ( li2024llmassistedstaticanalysisdetecting ) combine LLMs with static analysis, detecting 55 of 120 vulnerabilities and finding six new ones. Zhou et al. ( zhou2024comparisonstaticapplicationsecurity ) and Steenhoek et al. ( closingthegap2025 ) compared static analyzers and LLMs, finding higher precision but limited developer trust. All studies indicate that LLMs exhibit promising traits that encourage further extensive research in this domain. 3. Methodology 3.1. LLMs Selection We selected two groups of the latest released LLMs: code-specific models and general-purpose models . We examined three popular leaderboards for LLMs, such as Chatbot Arena LLM Leaderboard ( lmsys2024 ) , Trustbit LLM Benchmark ( trustbit2024 ) , and CanAiCode Leaderboard ( canaicode2024 ) , and cross-checked to select these LLMs from the top 20 open-source LLMs. Additionally, we chose each code-based LLM and its general version in order to conduct a comparative analysis of their performance. Table 2 summarizes the models and the versions

Read the complete text