CASP: An evaluation dataset for formal verification of C code

Niclas Hertzberg, Merlijn Sevenhuijsen, Liv Kåreborn, Anna Lokrantz · 2025

Read CASP: An evaluation dataset for formal verification of C code by Niclas Hertzberg, Merlijn Sevenhuijsen, Liv Kåreborn, Anna Lokrantz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18798.txt.

Introduction

Niclas Hertzberg, Merlijn Sevenhuijsen, Liv Kåreborn, and Anna Lokrantz present in 2025 a test collection for checking whether C programs truly meet their written specifications. Language models now write code quickly, but the code they produce comes with no firm promise of being correct, and checking it by hand is slow expert work.

Their dataset, CASP, pairs C code with formal specifications written in the ACSL language. Through a careful multi-stage filter, they drew five hundred six verified pairs from an existing source collection. The set gives researchers a shared yardstick for judging tools that claim to prove code correct.

Catalog record

CREATORNiclas Hertzberg, Merlijn Sevenhuijsen, Liv Kåreborn, Anna Lokrantz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18798

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CASP: An evaluation dataset for formal verification of C code Niclas Hertzberg, Merlijn Sevenhuijsen, Liv Kåreborn, Anna Lokrantz Abstract Recent developments in Large Language Models (LLMs) have shown promise in automating code generation, yet the generated programs lack rigorous correctness guarantees. Formal verification can address this shortcoming, but requires expertise and is time-consuming to apply. Currently, there is no dataset of verified C code paired with formal specifications that enables systematic benchmarking in this space. To fill this gap, we present a curated evaluation dataset of C code paired with formal specifications written in ANSI/ISO C Specification Language (ACSL). We develop a multi-stage filtering process to carefully extract 506 506 pairs of C code and formal specifications from The Stack 1 and The Stack 2. We first identify C files annotated with formal languages. Then, we ensure that the annotated C files formally verify, and employ LLMs to improve non-verifying files. Furthermore, we post-process the remaining files into pairs of C code and ACSL specifications, where each specification-implementation pair is formally verified using Frama-C. To ensure the quality of the pairs, a manual inspection is conducted to confirm the correctness of every pair. The resulting dataset of C-ACSL specification pairs (CASP) provides a foundation for benchmarking and further research on integrating automated code generation with verified correctness. 1 Introduction Large Language Models (LLMs) for code generation have achieved remarkable results in recent years, showing strong performance on tasks such as generating syntactically valid functions and providing code completions [ 7 , 8 , 20 , 23 ] . While LLMs demonstrate value across many coding tasks, their utility remains limited in domains with strict safety and quality requirements, such as safety-critical systems, where software failures can lead to severe consequences. A key limitation is that code generated by LLMs cannot be reliably guaranteed to be correct. In contrast to the probabilistic nature of LLMs, formal specification languages offer a potential solution to the nondeterministic behavior of LLMs. These formal specification languages provide robust means to specify program behavior, which can then be verified using formal verification tools. However, the adoption of such formal specification languages faces practical challenges: the pairs of formal specifications and associated code must be manually written, which is both time-consuming and requires expertise. Given that manually creating such pairs is time-consuming, LLMs provide a promising application of being able to generate code from specifications or vice versa. The task of generating specifications from code and code from specifications is arguably a most fundamental use case for applying LLMs to formal specification generation, directly targeting the most time-consuming tasks. Furthermore, like the related task of correcting non-verifying code, these tasks provide an easily interpretable success metric through formal verification since the generated pairs either verify or fail to verify. However, evaluating the feasibility and progress of such an approach requires ways to measure the performance of LLMs in the form of dedicated evaluation datasets. We present such a dataset, consisting of verified pairs, that is specifically designed to benchmark this core generative capability. Existing datasets containing ACSL specifications [ 25 , 1 , 2 , 11 , 4 ] and C code have two significant limitations. First, the datasets are limited in size and therefore lack the breadth needed to cover diverse real-world use cases. Lacking large and diverse datasets, researchers cannot draw general conclusions regarding LLMs’ abilities to generate formally verified pairs. Second, existing datasets are typically distributed as collections of whole C source files, where specifications are embedded as comments. This format requires a non-trivial parsing and extraction step to isolate individual specification-implementation pairs before they can be used in an evaluation pipeline for pair-generation tasks. These two limitations hinder progress in integrating formal methods into the software development process. Our work addresses this fundamental gap by providing a dataset with sufficient breadth and volume to give researchers a reliable benchmark. Our dataset consists of C code and formal specifications in ANSI/ISO C Specification Language (ACSL), chosen for its adoption by the Frama-C [ 10 , 18 ] verification platform, which is used in both academic research and industrial contexts for verifying critical properties of C programs [ 12 , 27 ] . We call this dataset CASP, short for C-ACSL specification pairs. We create CASP by first sourcing all C files from The Stack v1 and The Stack v2 repositories. The C files are then filtered using three steps: (1) identifying and retaining high-quality C files containing formal specification in ACSL; (2) ensuring that the ACSL-annotated C files formally verify, and attempting to correct these if they do not verify (3) extracting individual function implementations and their formal specifications to obtain function-specification pairs. This paper offers the following contributions: 1. We present CASP: a unique dataset of C code paired with associated formal specifications in ACSL, accessible at: https://huggingface.co/datasets/nicher92/CASP_dataset . The dataset contains 506 pairs. These pairs are systematically extracted from large-scale open-source datasets (The Stack v1 and v2) and are formatted as pairs in order to be amenable to LLM evaluation. 2. We share the complete files from which each pair was taken, accessible at: https://huggingface.co/datasets/nicher92/CASP_source_files . The number of files is much larger than previously available public datasets, offering significantly more data for training and evaluation. Additionally, the files – and by extension the pairs– are all "minimally complete" – meaning the files have no dependencies other than the standard C libraries. 3. We detail our filtering, verification, and post-fixing procedures, ensuring that each file, as well as each code-specification pair, formally verifies and is consistent with one another. This dataset fills a gap by providing a valuable resource for benchmarking and training LLMs on the task of specification generation from code and vice versa. By offering a conveniently formatted dataset consisting of verified, minimally complete pairs of specifications and associated code, our work supports the development of advanced tools for software verification by contributing to the creation of more reliable software systems. The rest of this paper is organized as follows. We first provide a brief overview of formal verification, in particular the ANSI/ISO Specification Language (ACSL) in Section 2 . Then in Section 3 , we review existing formal specification datasets and their limitations. Section 4 explains our data collection methodology, followed by our file verification process in Section 5 . We then describe how we divided the files into specification-function pairs Section 6 . Section 7 presents the composition and key statistics of the dataset, with a discussion and analysis in Section 8 . Finally, Section 9 presents our conclusions and suggests directions for future work. 2 Background This section provides background on the specification language and verification tools used in our dataset. In particular, we describe the ANSI/ISO C Specification Language (ACSL) and the Frama-C verification framework, with a focus on the WP and RTE plugins used to check correctness and runtime safety. In the dataset, we focus on ANSI/ISO C Specification Language (ACSL), which enables the formal verification of C code. The language is designed for use with the Frama-C verification framework, a framework for static analysis and deductive verification [ 3 ] . 2.1 ANSI/ISO C Specification Language ACSL is a contract-based specification language that allows

Read the complete text