Neurosymbolic Auditing of Natural-Language Software Requirements
Bethel Hall, William Eiers · 2026
Read Neurosymbolic Auditing of Natural-Language Software Requirements by Bethel Hall, William Eiers (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13817.txt.
Introduction
Bethel Hall and William Eiers show that large word models, fitted with an SMT solver, can audit software specs by turning them into formal logic. The team presents VERIMED, a neuro-symbol pipe running this idea for medical-device software specs. In counter-case-guided repair on a kidney-care Q and A test, SMT counter-cases raise verified hits from 55.4 percent to 98.5 percent. Over wide tests on open-source kidney-care safety specs, VERIMED reduces fuzz-prone specs and enables strict checks through SMT-based queries.
Catalog record
| CREATOR | Bethel Hall, William Eiers |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50769 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13817
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neurosymbolic Auditing of Natural-Language Software Requirements Bethel Hall, William Eiers Abstract Natural-language software requirements are often ambiguous, inconsistent, and underspecified; in safety-critical domains, these defects propagate into formal models that verify the wrong specification and into implementations that ship unsafe behavior. We show that large language models, equipped with an SMT solver, can audit such requirements: translating them into formal logic, detecting ambiguity through stochastic variation in the generated formalization, and exposing inconsistency, vacuousness, and safety violations through solver queries on the resulting specification. We present VeriMed , a neurosymbolic pipeline that operationalizes this idea for medical-device software requirements, and report two findings. First, stochastic variation across independent formalizations is a signal of ambiguity: requirements that admit multiple plausible interpretations produce SMT-inequivalent formalizations, and bidirectional SMT equivalence checking turns this disagreement into a solver-checkable test. Second, the usefulness of symbolic feedback depends on its granularity: in counterexample-guided repair on a hemodialysis question-answering benchmark, concrete SMT counterexamples raise verified accuracy from 55.4% to 98.5%. Over an extensive experimental evaluation on open-source hemodialysis safety requirements, we show that the LLM-based approach in VeriMed successfully reduces ambiguity-sensitive requirements and enables rigorous auditing of software requirements through SMT-based queries. 1 Introduction Autoformalization, the translation of natural-language requirements into machine-verifiable specifications, has advanced rapidly with large language models [ 28 ] . Yet for safety-critical software requirements, syntactic validity is not enough. A well-formed formal specification can be globally inconsistent, contain requirements whose trigger conditions are unreachable, or admit multiple plausible readings of the same natural-language source. These failures are invisible by construction: when no ground-truth formalization exists, there is no oracle for correctness. This is inherent to LLM-generated formalizations, where the model resolves ambiguity by defaulting to the most statistically common reading in its training distribution or omits constraints implicit in the source. This problem is especially acute in domains such as medical devices, where requirements specify alarm conditions, pump behavior, and safe parameter ranges. If the requirement set is inconsistent, or if the autoformalizer silently picks one reading among several, downstream verification can succeed while checking the wrong specification entirely. To address this, we propose VeriMed , a neurosymbolic framework that makes these failures detectable through two complementary checks. First, we audit the generated specification with solver queries that check global consistency, vacuousness, violatability, and redundancy at the requirement level, producing diagnostic flags that localize structural defects to specific requirements. Second, we detect and resolve ambiguity in the source text by exploiting the stochasticity of LLM-based autoformalizers as a diagnostic signal: sampling multiple independent SMT formalizations of the same requirement and comparing them with bidirectional SMT equivalence checking; persistent disagreement signals that the natural-language requirement admits more than one plausible reading. When defects are detected, we execute a targeted repair through a neurosymbolic feedback loop: solver feedback guides re-formalization for structural defects, and SMT-derived witnesses guide natural-language clarification for ambiguous requirements. The final requirement is then surfaced to the user for manual edits. Our contributions are as follows: • We introduce a neurosymbolic framework for analyzing natural-language software requirements. The system, VeriMed , autoformalizes requirements into a single SMT model and uses the solver’s output as the primary signal for downstream review. • We formulate four requirement-level SMT audits — global consistency, vacuousness, violatability, and redundancy — that operationalize established requirement-quality criteria. Applied to a published hemodialysis specification, the audits flagged 2 of 64 requirements as redundant. • We show that the quality of solver feedback determines the effectiveness of LLM repair. In requirement-grounded question answering, using the violated requirements as feedback alone raises verified accuracy from 55.4% (no feedback) to 80.0%; providing an SMT counterexample as additional feedback raises accuracy to 98.5%. • We introduce ambiguity detection via bidirectional SMT inequivalence across independently sampled formalizations, using solver-generated witnesses to drive clarification. To our knowledge, this is the first application of solver-checked sample disagreement to software requirements analysis. On the hemodialysis benchmark, the procedure flagged 12 of 64 requirements (18.8%) as producing multiple distinct encodings; all 12 converged to a single encoding after clarification. 2 Methodology 2.1 Problem Formulation Let R=\{r_{1},\dots,r_{n}\} be a set of natural-language software requirements. We translate R into a single SMT model M and use M to answer three classes of queries: requirement-level audits, query verification against scenario-based safety questions, and ambiguity detection over the source text. Encoding. Let T denote the translation from a requirement r_{i} to a formal encoding \rho_{i}=T(r_{i}) . Most safety requirements take a conditional form: “if condition P holds, then action Q must follow.” We encode each such requirement as \rho_{i}(\mathbf{x})\equiv p_{i}(\mathbf{x})\Rightarrow q_{i}(\mathbf{x}) , where \mathbf{x} denotes the typed state variables of the system (Boolean control states and real-valued device parameters) and p_{i},q_{i} are quantifier-free formulas over Booleans and linear real arithmetic. Invariant requirements, those that must hold unconditionally, are encoded directly as state predicates \rho_{i}(\mathbf{x}) with no antecedent. We also maintain a set of global domain constraints C(\mathbf{x}) encoding background facts shared across all requirements, such as flow rates being non-negative. The full model is M=C\wedge\bigwedge_{i=1}^{n}\rho_{i}. Audit queries. We define four solver queries over M that characterize the requirement set. 1. Global consistency asks whether M is satisfiable. If M is unsatisfiable, the requirements are mutually inconsistent, and an unsat core identifies a conflicting subset for repair. 2. Vacuousness applies to conditional requirements and asks whether the triggering condition p_{i}(\mathbf{x}) can ever hold under the domain constraints. We check this by asking whether C\wedge p_{i}(\mathbf{x}) is satisfiable. If it is unsatisfiable, r_{i} never applies and is therefore vacuous. Invariant requirements have no antecedent and are not subject to this check. 3. Violatability asks whether a requirement can be broken within the domain. We check C\wedge p_{i}(\mathbf{x})\wedge\neg q_{i}(\mathbf{x}) for conditional requirements and C\wedge\neg\rho_{i}(\mathbf{x}) for invariants; a satisfiable result returns a concrete violating assignment; an unsatisfiable result indicates that the violation is already ruled out by the domain constraints alone. 4. Redundancy asks whether M_{\setminus i}=C\wedge\bigwedge_{j\neq i}\rho_{j} already enforces r_{i} . We check M_{\setminus i}\wedge p_{i}(\mathbf{x})\wedge\neg q_{i}(\mathbf{x}) for conditional requirements and M_{\setminus i}\wedge\neg\rho_{i}(\mathbf{x}) for invariants; unsatisfiability in either case confirms redundancy, and the unsat core identifies the subsuming requirements. Query verification. Given a scenario-based safety question translated into a formal claim \phi , we ask whether \phi is supported by the requirements by checking the formula M\wedge\neg\phi . If it is unsatisfiable, M\models\phi and the claim is entailed; if it is satisfiable, the solver returns a counterexample showing a state consistent with M but inconsistent with \phi , which is then used as feedback for repair. Ambiguity detection. The previous queries treat M as fixed; we now consider whether M itself is well-determined by the source requirements. A natural-language requirement may admit more than one plausible reading, and the formalizer may silently commit to one. We define ambiguity operationally through agreement across independently sampled formalizations. Given two candidate encodings \rho_{i}^{a} and \rho_{i}^{b} of r_{i} , we say they agree if both C\wedge(\rho_{i}^{a}\wedge\neg\rho_{i}^{b}) and C\wedge(\rho_{i}^{b}\wedge\neg\rho_{i}^{a}) are unsatisfiable. If the first is satisfiable, the witnessing assignment exposes a state allowed by \rho_{i}^{a} but ruled out by \rho_{i}^{b} ; if the second is satisfiable, the witness exposes a state allowed by \rho_{i}^{b} but ruled out by \rho_{i}^{a} . We sample N formalizations of r_{i} and apply this pairwise