SpecPylot : Python Specification Generation using Large Language Models Conference: 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering; July 5–9, 2026; Montreal, QC, Canada 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 5–9, 2026, Montreal, QC, Canada DOI: 10.1145/3803437.3806427 ISBN: 979-8-4007-2636-1/26/07 CCS: Software and its engineering Formal software verification

Ragib Shahariar Ayon, Shibbir Ahmed · 2026

Read SpecPylot : Python Specification Generation using Large Language Models Conference: 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering; July 5–9, 2026; Montreal, QC, Canada 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 5–9, 2026, Montreal, QC, Canada DOI: 10.1145/3803437.3806427 ISBN: 979-8-4007-2636-1/26/07 CCS: Software and its engineering Formal software verification by Ragib Shahariar Ayon, Shibbir Ahmed (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16560.txt.

Introduction

Ayon and Ahmed present SpecPylot, a Python tool that makes working formal specs. Hand-written contracts are rare, so checking stays limited, and model-made specs often fail on syntax, strictness or wrong outputs. It offers trial contracts as icontract tags and checks them with crosshair symbol-based runs. When crosshair finds a failing case, only contracts change and the program stays as is. The tool also makes cover-driven pytest stubs and debug records, working for most programs despite bounded search limits.

Catalog record

CREATORRagib Shahariar Ayon, Shibbir Ahmed
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23203
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16560

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpecPylot : Python Specification Generation using Large Language Models Conference: 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering; July 5–9, 2026; Montreal, QC, Canada 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 5–9, 2026, Montreal, QC, Canada DOI: 10.1145/3803437.3806427 ISBN: 979-8-4007-2636-1/26/07 CCS: Software and its engineering Formal software verification Ragib Shahariar Ayon, Shibbir Ahmed Abstract . Automatically generating formal specifications could reduce the effort needed to improve program correctness, but in practice, this is still challenging. Many developers avoid writing contracts by hand, which limits the use of automated verification tools. Recent large language models (LLMs) can generate specifications from code, but these specifications often fail in terms of verification. The reason is syntax errors, overly strict constraints, or mismatches with program behavior. We present SpecPylot , a Python tool that synthesizes executable specifications for Python programs as icontract annotations and checks them using CrossHair ’s symbolic execution. The tool relies on LLMs to propose candidate contracts and uses CrossHair to validate them. When CrossHair finds a concrete counterexample, SpecPylot updates only the generated contracts and leaves the program itself untouched. In addition, the tool can produce coverage-driven pytest stubs and keep detailed execution artifacts that are useful during debugging. Overall, the evaluation indicates that SpecPylot is able to generate CrossHair -compatible contracts for most programs, but it also highlights the practical limits introduced by bounded symbolic exploration and differences in LLM behavior. † † cc-license: by 1. Introduction Writing software specifications is still a slow and error-prone process, even for experienced developers. In many Python projects, explicit specifications such as preconditions, postconditions, or invariants are missing altogether, which limits the applicability of automated verification and testing techniques in practice. This gap between available analysis tools and real-world code has been widely observed in prior work on specification inference and contract-based verification ( 8 ; 7 ) . Recent LLMs appear promising in this space. Given source code, they can often articulate high-level behavioral constraints that resemble human-written specifications, as demonstrated in several recent studies on LLM-assisted specification and invariant generation ( 14 ; 6 ; 18 ) . However, specifications produced directly by LLMs are frequently problematic. They may be inconsistent, overly restrictive, or incorrect regarding the program’s actual behavior. Furthermore, this issue was also reported in prior work evaluating LLM-generated specifications under formal verification ( 12 ; 11 ) . Without validation, such contracts are difficult to reuse or trust. We present SpecPylot , a contract synthesis and verification pipeline for Python that addresses this gap by combining LLM-based specification generation with solver-backed symbolic execution. SpecPylot first asks an LLM to produce executable contracts as icontract ( 15 ) annotations, and then relies on CrossHair to check whether those contracts are consistent with the program. If CrossHair finds a concrete counterexample, SpecPylot uses that information to trigger a refinement step. In that refinement step, the model is asked to revise the contracts while leaving the underlying code untouched. This design is inspired by recent verifier in the loop approaches for specification refinement, which were adapted to the Python ecosystem and executable contracts ( 12 ; 21 ) . At a high level, SpecPylot treats specification generation and specification checking as two separate concerns. The LLM is responsible for proposing candidate semantic constraints. However, CrossHair acts as an execution based sanity check that can either confirm those constraints or expose concrete violations within a bounded search. By iteratively refining contracts using counterexamples, SpecPylot improves specification quality without requiring manual intervention from the human developer. SpecPylot is implemented as a practical research tool rather than a mere prototype. It supports multiple LLM backends, configurable verification and refinement budgets, and optional coverage-driven test generation. Our evaluation shows that SpecPylot can generate checkable specifications for most programs in a small benchmark, while also highlighting the limitations imposed by bounded symbolic execution and the variability of LLM-generated contracts. In summary, this work makes the following contributions. • We designed and developed SpecPylot , a Python tool designed to generate executable specifications using LLMs, while also verifying them through symbolic execution. • Our approach allows the use of concrete counterexamples from CrossHair to refine and correct any incorrect or overly strict contracts, without changing the original program. • The system is built to be practical and user-friendly, supporting various LLM providers, offering user-defined limits for verification and refinement, and including optional test generation based on coverage analysis. • We evaluated the tool on a small set of Python programs, providing insights into which types of specifications can be verified, how long the process takes, and the costs involved. 2. Related Work/Research Gap Earlier work on automatic specification generation relied on program analysis to extract properties from code. More recent approaches use learning-based techniques, including large language models. SpecPylot builds on this direction by focusing on how LLM-generated specifications can be made usable under execution-based verification. Before LLMs were widely used, most work on specification inference relied on static or dynamic analysis. Dynamic invariant detectors such as Daikon ( 8 ) infer properties by observing execution traces, while static techniques based on constraint solving or abstract interpretation attempt to establish lightweight correctness guarantees ( 2 ; 7 ) . Several hybrid approaches combine these ideas with symbolic execution or mutation strategies ( 13 ; 6 ) . Although these methods can be effective in controlled settings, they often depend on restricted specification grammars, limited language features, or substantial manual effort. In practice, this makes them difficult to apply to realistic Python programs. More recently, researchers have explored using large language models to generate specifications directly from source code. This direction builds on the strong performance of LLMs in related tasks such as code generation ( 20 ) , summarization ( 1 ) , and defect prediction ( 10 ) . Several studies report that LLMs can produce invariants or contracts that resemble those written by developers ( 14 ; 6 ; 18 ) . SpecGen ( 12 ) , for example, combines few-shot prompting with mutation-based heuristics to generate JML specifications that can be checked by a verifier. Other approaches filter model outputs using ranking or static analysis. A common limitation across this line of work is that specification generation is typically treated as a one-shot activity, using fixed prompts or a single model, with little support for diagnosing or repairing incorrect specifications. Alongside generation techniques, some efforts concentrate on evaluation rather than generation. FormalBench ( 11 ) examines the robustness and consistency of LLM-generated specifications under different prompting strategies. Agent-based frameworks such as CodeVisionary ( 17 ) and RepoMasterEval ( 19 ) study collaborative reasoning in code generation more broadly. Recent work also explores verifier-in-the-loop approaches that use test oracles or formal feedback to refine generated annotations ( 9 ) . While these approaches show promising results in verification-aware languages such as Dafny, they target different ecosystems and rely on test assertions as primary specification oracles. In the Python ecosystem, the Python-by-Contract dataset ( 21 ) provides a collection of programs annotated with executable icontract specifications ( 15 ) and has been used to evaluate tools

Read the complete text