Grammar-Constrained Refinement of Safety Operational Rules Using Language in the Loop: What Could Go Wrong Conference: 21st International Conference on Software Engineering for Adaptive and Self-Managing Systems; April 13–14, 2026; Rio de Janeiro, Brazil 21st International Conference on Software Engineering for Adaptive and Self-Managing Systems (SEAMS ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3788550.3794884 ISBN: 979-8-4007-2445-9/2026/04 CCS: Software and its engineering Software safety

Khouloud Gaaloul, Zaid Ghazal, Madhu Latha Pulimi, Sam Emmanuel Kathiravan · 2026

Read Grammar-Constrained Refinement of Safety Operational Rules Using Language in the Loop: What Could Go Wrong Conference: 21st International Conference on Software Engineering for Adaptive and Self-Managing Systems; April 13–14, 2026; Rio de Janeiro, Brazil 21st International Conference on Software Engineering for Adaptive and Self-Managing Systems (SEAMS ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3788550.3794884 ISBN: 979-8-4007-2445-9/2026/04 CCS: Software and its engineering Software safety by Khouloud Gaaloul, Zaid Ghazal, Madhu Latha Pulimi, Sam Emmanuel Kathiravan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23523.txt.

Introduction

Gaaloul and colleagues build a loop that updates safety rules for machines that sense and act in the real world. As conditions change, rules must change too, but edits must obey the rule grammar and must not merely fit past data. Their loop pairs what-if reasoning with strict grammar checks to keep edits sound. Tested on self-driving control, it fixes a flaw in a rule from a standard baseline.

Catalog record

CREATORKhouloud Gaaloul, Zaid Ghazal, Madhu Latha Pulimi, Sam Emmanuel Kathiravan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33891
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23523

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Grammar-Constrained Refinement of Safety Operational Rules Using Language in the Loop: What Could Go Wrong Conference: 21st International Conference on Software Engineering for Adaptive and Self-Managing Systems; April 13–14, 2026; Rio de Janeiro, Brazil 21st International Conference on Software Engineering for Adaptive and Self-Managing Systems (SEAMS ’26), April 13–14, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3788550.3794884 ISBN: 979-8-4007-2445-9/2026/04 CCS: Software and its engineering Software safety Khouloud Gaaloul, Zaid Ghazal, Madhu Latha Pulimi, Sam Emmanuel Kathiravan Abstract . Safety specifications in cyber-physical systems (CPS) capture the operational conditions the system must satisfy to operate safely within its intended environment. As operating environments evolve, operational rules must be continuously refined to preserve consistency with observed system behavior during simulation-based verification and validation. Revising inconsistent rules is challenging because the changes must remain syntactically correct under a domain-specific grammar. Language-in-the-loop refinement further raises safety concerns beyond syntactic violations, as it can produce semantically unjustified refinements that overfit to the observed outcomes. We introduce a framework that combines counterfactual reasoning with a grammar-constrained refinement loop to refine operational rules, aligning them with the observed system behavior. Applied to an autonomous driving control system, our approach successfully resolved the inconsistencies in an operational rule inferred by a conventional baseline while remaining grammar compliant. An empirical large language model (LLM) study further revealed model-dependent refinement quality and safety lessons, which motivate rigorous grammar enforcement, stronger semantic validation, and broader evaluation in future work. † † cc-license: by 1. Introduction Safety specifications for cyber-physical systems (CPS) rely on operational rules that translate safety requirements into testable conditions on system executions for verification and validation. In autonomous driving, these rules are scoped by the Operational Design Domain (ODD) ( 6 ; 8 ) and informed by ISO 26262 ( 5 ) and ISO/PAS 21448 (SOTIF) ( 14 ) . They are typically expressed as logical or quantitative constraints over environmental and system variables, and evaluated through simulation-based testing ( 21 ) to ensure safe operation within the declared operational boundaries. However, as the operating conditions evolve, the observed system outcomes may shift and expose inconsistencies with the verdict of the operational rule. Maintaining rule validity and ensuring continued alignment with observed executions is a recurring challenge. This motivates an automated rule refinement process that restores consistency while preserving the underlying safety requirement and keeping the stated ODD unchanged. Mining operational rules from observed traces has commonly been performed by learning linear temporal logic (LTL) properties ( 20 ; 19 ) and by applying specification mining and machine learning (ML) methods ( 10 ; 16 ) , typically constrained by templates rather than a domain-specific grammar. Other works use surrogate models ( 12 ; 11 ; 24 ) and genetic programming (GP) ( 7 ) as interpretable methods to infer and correct STL properties, or apply parameter mining and falsification ( 1 ; 9 ; 15 ) to tune bounds that make observed behaviors satisfy a fixed specification. No prior work is designed to automatically refine an existing inconsistent operational rule under a domain-specific grammar. Recent work has explored using large language models (LLMs) to support the construction and maintenance of safety specification artifacts. Nouri et al. ( 18 ) propose a prompt-based pipeline for an automotive SafetyOps workflow that generates safety requirements and then checks the resulting rule set for redundancy, contradictions, and other quality issues. Li et al. ( 17 ) use LLMs to generate LTL specifications under safety restrictions, then iteratively refine candidate formulas using language inclusion checks and counterexamples. Both works commonly combine LLM generation with automated validation. However, to our knowledge, no prior work uses grammar-constrained LLMs to refine an existing inconsistent operational rule to align with observed system behavior. Moreover, validation in LLM-based specification synthesis pipelines typically targets syntactic or format compliance, rather than consistency with runtime outcomes. We propose an approach for refining operational rules to resolve inconsistencies between rule verdicts and observed system behavior. Refinement is restricted to operational rules that operationalize a fixed safety requirement within a stated ODD. Rather than constructing a formal proof of requirement satisfaction, our approach is guided by observed counterfactual evidence. The core novelty is combining counterfactual analysis with a grammar-constrained LLM guided by a domain-specific grammar to synthesize minimal refinements of operational rules while preserving syntactic correctness and semantic validity. The main contribution of this paper is a rule refinement framework that combines counterfactual reasoning to localize inconsistency boundaries between operational rules and observed behavior, and grammar-constrained LLMs to generate interpretable, syntactically valid refinements to the operational rules that capture a pre-specified safety requirement within a stated ODD. We report an initial empirical study on an autonomous driving subsystem (ADS), evaluating effectiveness against a conventional baseline. We further assess the model-dependent language-in-the-loop refinement quality across multiple LLM variants and derive safety lessons. Organization. The remainder of this paper is organized as follows. Section 2 presents the operational rule refinement framework, including the rule semantics, the domain-specific grammar, and the counterfactual guided, grammar constrained refinement loop. Section 3 reports our experimental exploration on an autonomous driving subsystem and summarizes the results of the LLM variant study. Section 4 concludes and outlines directions for future work. Figure 1. Operational Rule Refinement Approach Overview. Overview of the operational rule refinement pipeline showing inputs (rule set, grammar, labeled executions), a counterfactual step to localize inconsistency boundaries, a grammar constrained LLM refinement loop, and validation producing an updated rule. 2. Operational Rule Refinement Framework To address inconsistencies between operational rules and observed system behavior, we introduce a grammar-constrained rule refinement framework that leverages counterfactual reasoning combined with grammar-constrained LLM to produce interpretable, syntactically and semantically valid refinements of operational rules. The framework can be applied as a corrective mechanism for outdated operational rules within safety specifications that have become inconsistent with evolving system behavior. As a running example, we consider an autonomous driving controller parameterized by three scenario features; the ego vehicle speed ( ego_speed ), the longitudinal distance to the front vehicle ( dist_front ), and the lateral offset within the lane ( lane_offset ). In the context of autonomous driving, these features directly influence lane keeping behavior and collision risk. Operational rules encode admissible regions of the input space in which the unchanged safety requirement is intended to hold, i.e., minimum headway constraints at a given speed or allowable lane deviation under certain road conditions. Such rules capture the safety requirement into conditions over the variables, and may lose consistency with observed controller behavior as the operating environment changes or evolves. When these rules become inconsistent, they can yield misleading verdicts even if the underlying safety requirement is unchanged, motivating refinement of the rules rather than adaptation of the safety specification. 2.1. Operational Rules and Semantics Let X\subseteq\mathbb{R}^{d} denote the input domain of the system under test (SUT), where each input vector x=[i_{1},\ldots,i_{d}] represents input variables. The label of x is y , where y=\mathsf{Pass} if the observed system outcome satisfies the safety requirement, and y=\mathsf{Fail} if it violates the safety requirement. We assume these outcome labels reflect the SUT simulator ground truth for the underlying safety requirement, handling label noise is outside our refinement objectives. An operational rule is a predicate r:X\to\{\mathsf{true},\mathsf{false}\} that constrains the operating

Read the complete text