AutoReSpec : A Framework for Generating Specification using Large Language Models Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering (FORGE ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793655.3793731 ISBN: 979-8-4007-2477-0/2026/04 CCS: Software and its engineering Specification languages CCS: Software and its engineering Formal software verification CCS: Computing methodologies Natural language processing CCS: Computing methodologies Machine learning

Ragib Shahariar Ayon, Shibbir Ahmed · 2026

Read AutoReSpec : A Framework for Generating Specification using Large Language Models Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering (FORGE ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793655.3793731 ISBN: 979-8-4007-2477-0/2026/04 CCS: Software and its engineering Specification languages CCS: Software and its engineering Formal software verification CCS: Computing methodologies Natural language processing CCS: Computing methodologies Machine learning by Ragib Shahariar Ayon, Shibbir Ahmed (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03758.txt.

Introduction

Ragib Shahariar Ayon and Shibbir Ahmed built AutoReSpec, a system that writes formal software specifications with language models and repairs its own failures. It picks a model pair and prompt setup from the program's shape, then calls a partner model with validator feedback when the first attempt fails. On 72 Java programs it passed 67, beating SpecGen and FormalBench on success and completeness.

Catalog record

CREATORRagib Shahariar Ayon, Shibbir Ahmed
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55045
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03758

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AutoReSpec : A Framework for Generating Specification using Large Language Models Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering (FORGE ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793655.3793731 ISBN: 979-8-4007-2477-0/2026/04 CCS: Software and its engineering Specification languages CCS: Software and its engineering Formal software verification CCS: Computing methodologies Natural language processing CCS: Computing methodologies Machine learning Ragib Shahariar Ayon, Shibbir Ahmed Abstract . Formal specification generation has recently drawn attention in software engineering as a way to improve program correctness without requiring manual annotations. Large Language Models (LLMs) have shown promise in this area, but early results reveal several limitations. Generated specifications often fail verification due to syntax errors, logical inaccuracies, or incomplete reasoning, especially in programs with loops or branching logic. Techniques like SpecGen and FormalBench attempt to address this through prompting and benchmarking, but they typically rely on static prompts and do not offer mechanisms for recovering from failure or adapting to different program structures. In this paper, we present AutoReSpec , a collaborative framework that combines open and closed-source LLMs for verifiable specification generation. AutoReSpec dynamically chooses an LLM pair and prompt configuration based on the structure of the input program. If the primary LLM fails to produce a valid output, a collaborative model is invoked, using validator feedback to refine and correct the specification. This two-stage design enables both speed and robustness. We evaluate AutoReSpec on a new benchmark of 72 real-world and synthetic Java programs. Our results show that it achieves 67 passes out of 72, outperforming SpecGen and FormalBench in both Success Probability and Completeness. Our experimental evaluation achieves a 58.2% success probability and a 69.2% completeness score, while cutting evaluation time by 26.89% on average compared to prior methods. Together, these results demonstrate that AutoReSpec offers a scalable, efficient, and reliable approach to LLM-based formal specification generation. † † cc-license: by 1. Introduction Figure 1 . Motivating example illustrating Prior tools like SpecGen and FormalBench fail on the Java method TokenTest02 , achieving 0% success probability and completeness under OpenJML with Z3. In contrast, AutoReSpec ’s collaborative pipeline combining an open-source model with a proprietary fallback yields a correct, verifiable specification with 100% success and completeness, fewer validator calls, and faster runtime. A three-column comparison for the Java method \texttt{TokenTest02} showing outputs from SpecGen, FormalBench, and \name. The SpecGen and FormalBench columns include candidate JML specifications that fail OpenJML verification, while the \namecolumn shows a corrected JML specification that verifies successfully after collaborative refinement with verifier feedback, with accompanying summary metrics for success, completeness, validator calls, and runtime. Formal specifications play a foundational role in reasoning about program behavior. They enable a range of tasks in software engineering, including automated verification ( 16 ) , test generation ( 15 ) , and program synthesis ( 5 ) . Despite their theoretical value, formal specifications are largely absent in real-world codebases. Writing them by hand requires deep expertise in formal methods, familiarity with specification languages like JML ( 24 ) , and significant time investment, especially for programs with complex control flow or semantic constraints. Several works have explored ways to automate this process ( 17 ; 15 ; 30 ; 37 ) , with recent work focusing on leveraging LLMs to learn formal specifications directly from code. A number of LLM-based approaches have been proposed for this task. SpecGen ( 26 ) applies a few-shot conversational prompting and mutation-based specification generation to improve verification success. Pei et al. ( 34 ) investigate whether LLMs can synthesize inductive invariants with fine-tuned models. FormalBench ( 22 ) evaluates LLMs on a broader set of semantic reasoning tasks, using specification synthesis as a proxy for deeper program understanding. While these efforts demonstrate early promise, they face important limitations. SpecGen and Pei et al. lack generality across model families and fail to recover when initial specifications are unverifiable. FormalBench, by design, focuses on evaluation rather than generation, and does not investigate interactive or collaborative refinement workflows. None of these systems supports adaptive prompting or dynamic model selection tailored to program complexity. To address these challenges, we introduce AutoReSpec , a collaborative LLM framework for synthesizing formal specifications in Java Modeling Language (JML). Although AutoReSpec is conceptually language-agnostic, supporting any setting where code and specification pairs can be verified, our implementation targets Java and JML due to the availability of mature verification tooling and benchmark datasets. AutoReSpec is built on three key ideas: (1) dynamically selecting the optimal LLM pair (primary and fallback) and few-shot configuration based on the program type; (2) generating and refining prompts through a conversational loop guided by validator feedback; and (3) selectively invoking a collaborative model upon primary-model failure. AutoReSpec first runs the primary LLM in a validator-guided refinement loop; if it still fails within the refinement budget, the last candidate specification and verifier error are forwarded to the collaborative LLM for focused recovery. Developing such an end-to-end system poses several technical challenges. First, different programs require different prompt strategies and model capabilities. Second, verification failures require error-guided prompt refinement that is both specific enough to resolve issues and general enough to generalize across models. Third, integrating multiple LLMs and verifier iterations without excessive latency or token overflow requires careful prompt truncation, memory management, and recovery logic. AutoReSpec mitigates these challenges through an adaptive recommender for model selection, a validator-guided refinement engine for error correction, an efficient prompt management module for context control, and a collaborative fallback mechanism for recovery. Under the dynamic model selection configuration (RQ1), the collaborative version of AutoReSpec verified 119 out of 120 programs from SpecGenBench, achieving 69.32% success probability (SP) and 60.33 % completeness (C). On the full 72-program benchmark (RQ2) combining SpecGenBench, SV-COMP, and real-world OpenJML issues, AutoReSpec achieved 58.2% SP and 69.2% C, while reducing average generation time by 26.89% compared with prior tools. Validator logs further confirmed stable recovery behavior across diverse program types. Our contributions are as follows: • We propose AutoReSpec , a novel collaborative framework that recommends LLM-model pairs based on program structure and uses verifier-guided conversational refinement to generate verifiable specifications. • We design a collaborative prompt engineering strategy that combines few-shot examples, validator feedback, and automated prompt truncation to support long-context interactions and enable targeted error resolution. • We utilize a mutation-based completeness score, success probability, and number of passes, evaluation time to rank the LLMs for recommendation. • We construct AutoReSpecBench , a new benchmark combining real-world GitHub issues, loop-heavy SV-COMP programs, and challenging cases from SpecGenBench, and release a public leaderboard for reproducible evaluation ( 7 ) . • We release our full framework ( 9 ) , including code, dataset, and VS Code extension ( 11 ) , as an open-source tool chain to support future work in specification generation and LLM-based reasoning. The remainder of this paper is organized as follows. Section 2 presents a motivating example that highlights the key challenges in automated specification generation. Section 3 details the design of the AutoReSpec framework. Section 4 describes the experimental setup, while Section 5 reports the results and analysis. Section

Read the complete text