How Do Solidity Versions Affect Vulnerability Detection Tools? An Empirical Study

Gerardo Iuliano, Davide Corradini, Michele Pasqua, Mariano Ceccato, Dario Di Nucci · 2025

Read How Do Solidity Versions Affect Vulnerability Detection Tools? An Empirical Study by Gerardo Iuliano, Davide Corradini, Michele Pasqua, Mariano Ceccato, Dario Di Nucci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05515.txt.

Introduction

The authors ask how language versions affect smart contract security tools in a 2025 study. Ethereum contracts are written in a fast-evolving language. Each version changes how code compiles and behaves. Detection tools may silently degrade as versions advance.

Twenty-five hundred real contracts formed a balanced wild data set. A separate set carried deliberately injected flaws. A standard framework was extended with eleven more tools. Crash resilience, detection quality, and runtime were all measured. Most tools survived early language versions without crashing. Only a fifth had been updated for newer versions. Detection performance varied noticeably across versions. Language changes, not tool bugs alone, explained the drift. Execution time also shifted by version. The study frames a general software analysis problem. Tools must be re-validated as languages evolve. Auditors cannot assume yesterday's accuracy persists.

Catalog record

CREATORGerardo Iuliano, Davide Corradini, Michele Pasqua, Mariano Ceccato, Dario Di Nucci
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-05515

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Do Solidity Versions Affect Vulnerability Detection Tools? An Empirical Study Thanks: Gerardo Iuliano, Davide Corradini, Michele Pasqua, Mariano Ceccato, Dario Di Nucci Abstract Context: Smart contract vulnerabilities pose significant security risks for the Ethereum ecosystem, driving the development of automated tools for detection and mitigation. Smart contracts are written in Solidity, a programming language that is rapidly evolving to add features and improvements to enhance smart contract security. New versions of Solidity change the compilation process, potentially affecting how tools interpret and analyze smart contract code. Objective: In such a continuously evolving landscape, we aim to investigate the compatibility of detection tools with Solidity versions. More specifically, we present a plan to study detection tools by empirically assessing (i) their compatibility with the Solidity pragma directives, (ii) their detection effectiveness, and (iii) their execution time across different versions of Solidity. Method: We will conduct an exploratory study by running several tools and collecting a large number of real-world smart contracts to create a balanced dataset. We will track and analyze the tool execution through SmartBugs, a framework that facilitates the tool execution and allows the integration of new tools. I Introduction Smart contracts are the leading technology for developing blockchain-based decentralized applications that enable programmable and autonomous interactions on the blockchain. However, their adoption has been accompanied by significant security risks due to vulnerabilities that can be exploited, often leading to severe financial losses. Consequently, researchers and developers have prioritized the creation of vulnerability detection tools to protect the Ethereum ecosystem. The development of static and dynamic analysis tools has increased significantly in recent years [ 1 , 2 ] , offering automated approaches to identify potential defects in smart contracts. Tools such as Mythril [ 3 ] , Slither [ 4 ] , and Oyente [ 5 ] are widely used to detect vulnerabilities by analyzing the contract code. The Solidity programming language used to code Ethereum smart contracts has evolved significantly since its inception. Each version introduces new features, enhances security mechanisms, and deprecates outdated functionalities to address vulnerabilities and improve contract security and reliability. For example, version 0.5.x introduced the keyword constructor , while 0.8.x introduced natively checked arithmetic operations. Although essential to advance contract security, these updates often pose compatibility challenges. ⬇ 1 pragma solidity 0.8.0; 2 contract ExampleDC { 3 function f () public pure { 4 uint x = 0; 5 x --; // Revert on underflow 6 } 7 } 8 9 pragma solidity >=0.4.0 <0.8.0; 10 import "@openzeppelin/contracts/SafeMath.sol" ; 11 contract ExampleSM { 12 using SafeMath for uint ; 13 function f () public pure { 14 uint x = 0; 15 x = x . sub (1); // Revert on underflow 16 } 17 } Fig. 1: Underflow Check on Different Solidity Versions. Figure 1 shows two implementations for decreasing a variable: one with pragma 0.8.x and one with pragma < 0.8.0. Both are affected by underflow but handle it differently. In Solidity 0.8.x., arithmetic operations are checked by default [ 6 ] . Whatever tool you use on ExampleDC contract, it will detect an underflow and a false positive. In the ExampleSM contract with Solidity pragma >= 0.4.0 < 0.8.0, the OpenZeppelin SafeMath library provides a sub() function to handle the underflow caused by variable decrease. Solidity updates may create a critical need to evaluate the compatibility of detection tools with the evolving Solidity language. Consider Durieux et al. [ 7 ] as an example. The study features two tools, namely Mythril [ 3 ] and Oyente [ 5 ] , that significantly impact the results, particularly for vulnerabilities related to arithmetic operations. Oyente [ 5 ] was released in October 2016 and was designed to analyze contracts deployed on the blockchain until May 5, 2016, meaning it was validated on Solidity 0.3.x or earlier. However, Solidity 0.4.x was introduced in September 2016, introducing major changes such as mandatory function keywords, visibility modifiers, and the constructor keyword. One of the most significant updates was the release of the SafeMath library, which changed how developers implement arithmetic operations. Durieux et al. [ 7 ] conducted their experiments using the SmartBugs dataset, which primarily contains contracts written in Solidity 0.4.x and a few in 0.5.x. This discrepancy may explain the poor performance of certain tools, as false positives could arise from compatibility issues. Tools evaluated on older versions of Solidity without explicitly stating their compatibility with Solidity versions may lead developers and auditors to misuse them. Tool compatibility with the latest Solidity versions is a critical issue. However, most tools do not specify which Solidity versions they support, leading users to assume broad compatibility. For example, tools such as EVM-Shield [ 8 ] , Dfier [ 9 ] , sGuard+ [ 10 ] , LENT-SSE [ 11 ] , ContractCheck [ 12 ] , and PrettySmart [ 13 ] , published in 2024, were evaluated using the SmartBugs dataset. This dataset consists mainly of Solidity 0.4.x contracts, with only a few from Solidity 0.5.x. Meanwhile, Solidity’s latest major release, 0.8.x, was introduced in December 2020. None of these tools explicitly state their compatibility with specific Solidity versions. Another motivation comes from a preliminary analysis of the datasets collected by Iuliano et al. [ 2 ] , whose results show that the most frequent Solidity version used to evaluate the tools is 0.4.x, while the current language version is 0.8.x. To our knowledge, tools evaluated on older Solidity versions do not necessarily fail on newer versions, but their effectiveness remains uncertain. This uncertainty also motivates our investigation into tool compatibility because developers and auditors often use them. According to the survey by Chaliasos et al. [ 14 ] , 92% of participants report that their organization uses open-source tools. Their surveys reveal that 59% of developers and 48% of auditors use at least one tool to detect logic-related vulnerabilities. Having a complete idea of how tools behave on the various versions of Solidity can help practitioners select the best tools according to their needs, maximizing detection rate or minimizing execution time. We aim to determine when and why certain tools remain effective as Solidity evolves and identify which vulnerability categories are most affected by Solidity’s changes. Our study will focus on analyzing (i) the compatibility of detection tools with various Solidity versions, (ii) their performance in identifying vulnerabilities in real-world datasets and the changes introduced into a version that impacts them, and (iii) their execution time to provide a comprehensive understanding of the capabilities and limitations of detection tools in the current landscape of smart contract security. We will employ exploratory research that combines empirical analysis with structured evaluation to achieve our goals. We plan to build a dataset of real-world smart contracts and use the SmartBugs framework [ 15 , 16 ] , which supports the execution and integration of the tools, and evaluate their compatibility, performance, and execution time. The paper is structured as follows. Section II provides the background to understand the paper, reviewing related work and key concepts. Section III describes the methods used in the study, detailing the experimental setup, data collection, and procedures to ensure transparency and reproducibility. Section V discusses the threats to validity, addressing

Read the complete text