LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs
Luis Ibanez-Lissen, Lorena Gonzalez-Manzano, Jose Maria de Fuentes, Nicolas Anciaux · 2025
Read LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs by Luis Ibanez-Lissen, Lorena Gonzalez-Manzano, Jose Maria de Fuentes, Nicolas Anciaux (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24451.txt.
Introduction
Ibanez-Lissen and co-authors use a cheap probe to predict whether a shrunk language model will still catch software flaws. Fine-tuning compressed models on vulnerability data costs heavy computation, so knowing in advance which layers can be cut would save real money. Linear probes, tiny classifiers fitted on internal activations, supply that early reading in about two minutes of compute.
The probes are applied to two models hunting twelve of the most dangerous weakness types, across 480,000 C and C++ samples. They show a third to nearly three quarters of layers can go with no precision loss. They also forecast the final model's accuracy closely. The resulting compact models beat the state of the art at 86.9 percent accuracy, and one compressed model even tops its full-size parent while training faster.
Catalog record
| CREATOR | Luis Ibanez-Lissen, Lorena Gonzalez-Manzano, Jose Maria de Fuentes, Nicolas Anciaux |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24451
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMs Luis Ibanez-Lissen, Lorena Gonzalez-Manzano, Jose Maria de Fuentes, Nicolas Anciaux Abstract Large Language Models (LLMs) are being extensively used for cybersecurity purposes. One of them is the detection of vulnerable codes. For the sake of efficiency and effectiveness, compression and fine-tuning techniques are being developed, respectively. However, they involve spending substantial computational efforts. In this vein, we analyse how Linear Probes (LPs) can be used to provide an estimation on the performance of a compressed LLM at an early phase – before fine-tuning. We also show their suitability to set the cut-off point when applying layer pruning compression. Our approach, dubbed L P A S S LPASS , is applied in BERT and Gemma for the detection of 12 of MITRE’s Top 25 most dangerous vulnerabilities on 480k C/C++ samples. LPs can be computed in 142.97 s. and provide key findings: (1) 33.3 % and 72.2% of layers can be removed, respectively, with no precision loss; (2) they provide an early estimate of the post-fine-tuning and post-compression model effectiveness, with 3% and 8.68% as the lowest and average precision errors, respectively. L P A S S LPASS -based LLMs outperform the state of the art, reaching 86.9% of accuracy in multi-class vulnerability detection. Interestingly, L P A S S LPASS -based compressed versions of Gemma outperform the original ones by 1.6% of F1-score at a maximum while saving 29.4 % and 23.8% of training and inference time and 42.98% of model size. 1 Introduction The wide variety of devices, systems and networks favors the emergence of vulnerabilities, defined as weaknesses in a system or security control 1 1 1 https://csrc.nist.gov/glossary/term/vulnerability , last access September 2024 . Lots of research works have been carried out to counter them. Particularly, multiple works have focused on vulnerability detection [ 1 , 2 , 3 ] , that is, whether a piece of software is vulnerable or not, whereas others aim to identify them, that is, which vulnerability is present [ 4 , 5 , 6 ] . This latter problem is more challenging as it requires a fine-grained knowledge of all vulnerabilities to tell them apart [ 7 ] . Therefore, in this paper we concentrate on this issue. Context. The use of Artificial Intelligence (AI) has been a constant in this field. Many works apply traditional AI algorithms such as support vector machines [ 8 , 9 ] , K-nearest neighbor [ 8 , 9 ] or graph neural networks [ 10 , 11 ] , once having extracted code features like the program dependency graph [ 10 , 11 ] , the control flow graph [ 12 ] or the number of lines of code [ 8 ] , among others. However, since the appearance of Large Language Models (LLMs), which are based on using deep learning techniques to manage massive amounts of data, novel vulnerability detection systems have appeared [ 13 ] . The main advantage of these models is their ability to manage complex tasks, but the required computational cost becomes a burden. Some of these models apply millions or even billions of parameters whose tuning and execution involves a high cost in terms of time and resources. New research lines try to reduce LLMs, speeding up the classification process and reducing the amount of used computing resources. Techniques such as pruning focus on reducing the amount of model parameters [ 14 ] or network components [ 15 ] . On the other hand, quantization is another well-known approach that reduces the size of each parameter. However, compression methods incur in accuracy losses which have to be minimized [ 16 ] . In the context of vulnerability detection, to the best of authors’ knowledge only [ 17 ] , [ 18 ] have applied model compression. They apply knowledge distillation [ 19 ] to compress CodeBERT and GraphCodeBERT, reaching promising inference speeds and efficiency. However, they focus on a binary classification problem (i.e., vulnerable / non-vulnerable) with an accuracy of 59.9 % – only 10% over a random guess. Motivation. Efficiency in vulnerability detection is of outmost relevance considering the increasing pace of software generation. For the sake of illustration, Google Play counts on 2.61 billion apps nowadays 2 2 2 https://www.businessofapps.com/data/google-play-statistics/, last access 20 june 2024. . At the same time, not only the amount of vulnerabilities, but also their severity, have been steadily growing in the last decade. According to CVEdetails 3 3 3 https://www.cvedetails.com/, last access September 2024 , the amount of vulnerabilities with a severity ranked between 7 and 10 raised from 1.9k in 2014 to 16.7k in 2023. Therefore, a vulnerability detection mechanism streamlined with the publication process would be desirable. Interestingly, identifying which is the vulnerability at stake enables providing a suitable response – while dangerous vulnerabilities may require stronger controls, irrelevant ones may simply raise a warning before distributing the analysed pieces of software. As noted in [ 20 ] , fine-tuning of models involves substantial energy expenses. For the sake of illustration, pre-training and fine-tuning Meta’s LLaMA model could cause up to 2.76 MtCO 2 -eq emissions, equivalent to the total pollution caused by manufacturing one dose of COVID-19 vaccine for all humans on Earth [ 21 ] . Therefore, saving resources related to the model fine-tuning and compression is paramount. The closest effort to ours, namely Chen et al. [ 22 ] is focused on compressing visual models, thus unrelated to vulnerability detection. They proposed using feature representations in convolutional neural networks to identify layers with high weight overlap after training , aiming to reduce the model size. In contrast, our approach seeks to compress pretrained LLMs by determining which layers of a model provide valuable information for a specific task before any fine-tuning or further training . Therefore, although Chen et al.’s method can save resources post-training, it requires training the entire model beforehand. Our approach, on the other hand, reduces the model size prior to fine-tuning, minimizing the computational resources needed for that task. Research question and contribution. Given the computational cost of model fine-tuning and compression, our research question is: Can we predict their impact to make informed decisions on their use and save resources ? To address this, we leverage linear classifier probes (LPs) [ 23 ] to gain early insights of the internal model status that can be valuable to guide the use of fine-tuning and compression techniques and estimate their effect in performance. Our approach, dubbed LPASS (LPs As Stepping Stones) is inspired by recent efforts that apply LPs to analyse the knowledge captured at varying depths within LLMs [ 24 ] , as well as the internal uncertainty [ 25 ] . LPs are a first layer of explainability by providing interpretability. As such, our use of LPs can be regarded as a first step for a explainable compression method, which has been recently pointed out as an open issue [ 26 ] . The list of contributions is as follows: 1. We adopt linear probes (LPs) in vulnerability detection for 1) determining the cut-off point when applying layer