An Empirical Study of LLM-Generated Specifications for VeriFast
Wen Fan, Minh Tran, Sanya Dod, Xin Hu, Marilyn Rego, Danning Xie, Jenna DiVincenzo, Lin Tan · 2026
Read An Empirical Study of LLM-Generated Specifications for VeriFast by Wen Fan, Minh Tran, Sanya Dod, Xin Hu, Marilyn Rego, Danning Xie, Jenna DiVincenzo, Lin Tan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26490.txt.
Introduction
Wen Fan and seven colleagues test whether language models can write formal specifications for a program verifier. Verifiers prove that code behaves correctly, but humans must write the specifications they check against. For VeriFast, a verifier based on separation logic, those specifications are complex and slow to write. Separation logic reasons about how programs use memory. Earlier model-based help mostly targeted simpler verifiers.
Coverage spanned three hundred three C functions. The authors tried eight prompting approaches, ten language models, and three input types. Models preserved correct behaviour in code and specifications more than nine times in ten. Full verification success was much lower, at under one third of cases. Stronger models and formal contracts in the prompt helped. Most errors, the great majority, came from missing domain knowledge about separation logic verifiers.
Catalog record
| CREATOR | Wen Fan, Minh Tran, Sanya Dod, Xin Hu, Marilyn Rego, Danning Xie, Jenna DiVincenzo, Lin Tan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26490
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Study of LLM-Generated Specifications for VeriFast Wen Fan, Minh Tran, Sanya Dod, Xin Hu, Marilyn Rego, Danning Xie, Jenna DiVincenzo, Lin Tan Abstract Static verification tools can assure industrial scale software, but require significant human labor to write specifications. This is particularly true of static verifiers based on separation logic (SL verifiers), which excel at verifying heap-manipulating programs, but require many complex auxiliary specifications to reason about heap structure. Recent work applies large language models (LLMs) to generate code, tests, and proofs, including specifications for verifiers, but mostly targeting non-SL verifiers. To address this gap, this paper thoroughly evaluates how well LLMs perform when prompted to generate specifications for verifying 303 C functions with the SL verifier VeriFast. We explored eight prompting approaches, ten LLMs, and three input types in two stages. Quantitative and qualitative analyses are used to assess the LLM-generated code and specifications for functional behavior, verifiability and errors. The results show that LLMs preserve functional behavior in source code and specifications (both over 91%), but achieve modest verification success (31.4%). Using Gemini 2.5 Pro and providing formal contracts lead to higher success rates in our setting. Moreover, most errors (94%) come from LLMs’ mistakes in the domain-specific knowledge of SL verifiers such as VeriFast. These findings provide guidance for optimizing LLM-generated specifications for SL verifiers. I Introduction Advancements in satisfiability modulo theories (SMT) solvers [ 44 ] have driven the development of automated deductive verification tools—also known as auto-active, static, Hoare-logic [ 22 ] , or design-by-contract verifiers. Such verifiers work by taking specifications written in a first-order mathematical logic on program components ( e.g. pre- and postconditions on functions), which denote the intended program behavior, and then statically analyzing the program to prove its adherence to the specifications. The analysis builds up proof obligations ( e.g. at the end of a function the analysis’s data should imply the postcondition) that are sent to an SMT solver to discharge. Successful verification of a program means that it will not violate its specifications at run time. Mature static verifiers, such as Dafny [ 33 ] and VeriFast [ 32 ] , have been successfully applied in industry. Dafny was used to verify the functional correctness of Amazon Web Services’ (AWS) authorization engine [ 6 ] , and VeriFast was used to verify the absence of safety errors ( e.g. dividing by zero or illegal memory accesses) in two smart card applets, a Linux device driver, and an embedded Linux network management component [ 51 ] . While promising, static verifiers have not achieved the ideal in automation where users only need to specify their intended behavior for code. Instead, verifiers require auxiliary specifications, such as loop invariants, inductive lemmas, and folds and unfolds of predicates to make up for limitations in proof automation. Furthermore, writing these specifications requires internal knowledge of the verifier’s analysis and its status throughout the verification process. It should not come as a surprise then that it took 4 expert person-years to verify the AWS authorization engine [ 6 ] . In response, and with the success of large language models (LLMs) at generating code [ 7 , 56 ] , test-cases [ 68 , 72 ] , program invariants [ 16 , 50 , 73 , 61 , 5 ] , and proofs in proof assistants [ 82 , 77 , 26 , 70 , 17 ] , there has been a surge of recent work (all in the last 3 years) using LLMs to successfully generate specifications for such verifiers [ 57 , 71 , 25 , 28 , 41 , 43 , 45 , 49 , 58 , 31 , 52 , 78 , 8 , 75 , 53 , 1 , 59 ] . Only one of these works [ 53 ] concerns static verifiers based on separation logic [ 54 , 48 ] (SL verifiers), such as Viper [ 46 ] , Gillian [ 19 ] , and VeriFast [ 23 ] . This understudy is unfortunate as SL verifiers in particular excel at verifying both the functional and safety properties of heap-manipulating programs at scale [ 51 ] , but they suffer from requiring auxiliary specifications that are further complicated by reasoning about the heap. This leads to specification patterns that are unique to SL verifiers, e.g. numerous folds and unfolds, loop invariants, and lemmas breaking down or connecting predicates describing parts of the heap, and are worth studying as a result. Therefore, this paper seeks to empirically and thoroughly evaluate how well existing LLMs perform when prompted to generate specifications for verifying C programs by the SL verifier VeriFast [ 23 ] . In particular, we examine both their ability to preserve functional behavior and produce verifiable specifications across settings, and analyze the errors that arise when verification fails. To conduct the study, we develop a dataset of 303 functions from 60 C programs with three different input versions of each function containing their functional behavior specified in natural language or VeriFast’s specification language. Our dataset also includes fully specified and verified (ground truth) versions of these functions. Throughout our study, LLMs are prompted to generate missing specifications needed to verify a function given its input files; the resulting output files are then quantitatively and qualitatively analyzed. In particular, our study proceeds in two stages: 1) we conduct pilot studies on 45 functions to select the best performing prompting approach out of eight and three best performing LLMs out of ten, and then 2) we conduct a full study on the full dataset with the selected prompt approach and LLMs. For the pilot studies, we make selections based on verification success rates, the number of verification errors and their severity, and the number of functional behavior changes made to function pre- and postconditions and source code. In the full study, we manually code source code and pre- and postconditions in the output files to assess their functional behavior. We also calculate the percentage of functions passing verification with VeriFast. Finally, we sample 50 failing functions and manually record and fix errors until the function is verified. During this process, we record the error severity and the fix applied. We find that LLMs largely preserve functional behavior in both source code (91.0%) and pre- and postconditions (92%), but have a fairly modest verification success rate at 31.4%. Moreover, we find Gemini 2.5 Pro and providing stronger pre- and postconditions in the input achieve the best performance for verifiability. Lastly, our error analysis shows that the majority of errors (94%) arise from limitations in the knowledge of SL verifier ( e.g. syntax and heap reasoning), rather than general logical reasoning. We also provide actionable insights from our results, such as the suggestion to provide better heap related debugging information from a SL verifier to LLMs when verification fails and enabling the auto-feature of VeriFast–both to reduce verification errors and improve success rates. In summary, this paper makes the following contributions: • The first empirical and thorough study of LLMs’ ability to generate specifications for program verification with an SL verifier using prompt engineering. In particular, we consider 8 different prompting approaches, 10 different LLMs, and 3 different