Tool for Supporting Debugging and Understanding of Normative Requirements Using LLMs
Alex Kleijwegt, Sinem Getir Yaman, Radu Calinescu · 2025
Read Tool for Supporting Debugging and Understanding of Normative Requirements Using LLMs by Alex Kleijwegt, Sinem Getir Yaman, Radu Calinescu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05504.txt.
Introduction
Alex Kleijwegt, Sinem Getir Yaman, and Radu Calinescu present a tool that helps people understand normative requirements, in a two thousand twenty-five paper. Normative requirements capture social, legal, ethical, empathetic, and cultural norms a system must obey. Stakeholders like lawyers and ethicists help write them, often without technical training.
Formal methods can check such rules for consistency, but their output baffles non-technical users. Counterexamples from model checkers are hard to read. The tool, called SLEEC-LLM, uses large language models to explain those counterexamples in plain language. That makes the cycle of drafting and validating rules faster. The authors aim to keep formal rigor while opening the process to everyone involved.
Catalog record
| CREATOR | Alex Kleijwegt, Sinem Getir Yaman, Radu Calinescu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05504
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Tool for Supporting Debugging and Understanding of Normative Requirements Using LLMs Alex Kleijwegt, Sinem Getir Yaman, Radu Calinescu Abstract I Introduction Normative requirements specify social, legal, ethical, empathetic, and cultural (SLEEC) norms that must be observed by a system. To support the identification of SLEEC requirements, numerous standards and regulations have been developed (e.g., [ 1 ] ). These requirements are typically defined by stakeholders in the non-technical system with diverse expertise (e.g., ethicists, lawyers, social scientists). Hence, ensuring their consistency and managing the requirement elicitation process are complex and error-prone tasks [ 2 , 3 ] . Recent research has addressed this challenge using domain-specific languages to specify normative requirements as rules, whose consistency can then be analyzed with formal methods [ 4 , 5 , 6 , 7 ] . Nevertheless, these approaches often present the results from formal verification tools in a way that is inaccessible to non-technical users. This hinders understanding and makes the iterative process of eliciting and validating these requirements inefficient in terms of both time and effort. To address this problem, we introduce SLEEC-LLM, a tool that uses large language models (LLMs) to provide natural-language interpretations for model-checking counterexamples corresponding to SLEEC rule inconsistencies [ 5 , 4 ] . SLEEC-LLM improves the efficiency and explainability of normative requirements elicitation and consistency analysis. To demonstrate its effectiveness, we summarise its use in two real-world case studies involving non-technical stakeholders. II Problem Description Our tool operates on SLEEC rules specified in SLEEC DSL, a domain-specific language for normative requirements [ 4 ] that has proven accessible to stakeholders without a technical background (e.g., lawyers, ethicists, and regulators) [ 5 ] . The specification of a SLEEC rule set in SLEEC DSL comprises two parts: definitions and rules. Definitions declare events and measures corresponding to system capabilities, and to activities during system interactions with the environment, including humans. Events correspond to instantaneous actions, whereas measures represent capabilities to provide (immediately) information captured by values of data types, such as Boolean , numeric , and scale . Rules have the basic form “ when trigger then response ”. Such a rule defines the required response when the event in the trigger happens, and its conditions on measures, if any, are satisfied. A rule can be accompanied by one or more defeaters , introduced using the “ unless ” construct. The language incorporates time constructs that allow responses with deadlines and timeouts using the “ within ” construct. Given such a SLEEC rule set, an existing tool called SLEECVAL [ 4 ] translates these rules into a formal language called tock CSP , a time-variant process algebra, as illustrated in Fig. 1 . The formal semantics defined in CSP are then analyzed for consistency using the FDR model checker [ 8 ] . However, the results (particularly counterexamples) are difficult to interpret for non-technical stakeholders, e.g., the rules “ R1 when DetectUserFallen then CallEmergencySupport within 2 minutes unless emergencyLevel>L4 then CallEmergencySupport ” and “ R2 when DetectUserFallen and emergencyLevel<L2 then not CallEmergencySupport within 2 minutes ” are conflicting, and the tool reports a counterexample in the following form where the tock is a time unit. ⬇ < DetectUserFallen , emergencyLevel . E1 , emergencyLevel . E1 , tock , ..., tock > Such output is not easily understandable for non-technical users and lacks interpretation of exploratory resolution scenarios for the system, e.g., what type of error is occurring. Our LLM-enabled debugging and resolution tool (publicly available on GitHub [ 9 ] ) addresses this problem. III Tool Architecture and Workflow Prompt engineering and LLMs. Fig. 1 : SLEEC-LLM workflow. Developing an effective prompt for an LLM is an iterative and flexible process that significantly impacts the quality of the LLM’s output. The prompt evolves continuously, adapting to changes in input parameters like the number of counterexamples, available context, and the specific LLM model used. The prompt begins by providing the LLM with comprehensive context including (Fig. 1 ): (1) SLEEC rules specified in SLEEC DSL for understanding outputs through RAG (Retrieval Augmented Generation), (2) formal semantics in CSP, (3) counterexamples, and (4) a detailed natural-language description of the system. Following this context, the prompt includes a JSON file for the expected output, which is presented in Listing 1 . The output is expected to include the errors and two different suggestions for resolution. IV Preliminary Results A user study was conducted to evaluate whether the SLEEC LLM tool helps resolve ruleset conflicts more quickly and with fewer development iterations. Using a control study among SLEEC participants (experts in law, ethics and psychology) who completed two tasks for two case studies. The first group of experts used the SLEEC inconsistency resolution tool with the LLM for the ALMI case study, which involves a healthcare robot. The second group performed the same task for ASPEN, a forest-monitoring drone, also using the LLM tool. Listing 1: Output template for conflicting rules (partial) ⬇ " Conflicting Rules ":{ " Error ": { " Rule1 ": { name of Rule1 }, " Rule2 ": { name of Rule2 }, " Scenario ":{ a scenario that creates conflict in the rules }, " Category ": { deadlock , divergence , naming } " Justification ": { your justification } } " Resolution ": { add rule , combine rule , remove rule , modify rule " Suggestion1 ": { name of Rule1 A }, " Justification ": { your justification } " Suggestion2 ": { name of Rule2 }, " Justification ": { your justification } } } After completing their initial tasks, the groups switched case studies and repeated the tasks without the LLM tool. The tasks required participants to modify intentionally inconsistencies in SLEEC rulesets. Participants used the SLEEC tool to identify and fix rule conflicts iteratively, with a 25-minute time limit per task. When the LLM was used, participants could submit counterexample traces for suggestions. Quantitative metrics included: 1) time to resolve conflicts (from counterexample to rule fix), 2) number of iterations (SLEEC submissions until the ruleset was conflict-free) The results are summarised in Table I . Task Elapsed Time Resolved Rules Iterations Study 1: ALMI – Manual Time expired 2 / 4 6 Study 2: ASPEN – Manual Time expired 2 / 4 8 Study 1: ALMI – LLM assisted 12m 30s 4 / 4 4 Study 2: ASPEN – LLM assisted 25 m 4 / 4 6 TABLE I : Results of user study on rule conflict resolution (4-rule set) V Discussion and Related Work We present a tool to support SLEEC experts in debugging and understanding inconsistencies in normative requirements during the requirement validation process. Existing research has focused on consistency validation methods [ 5 , 6 ] , and large language models (LLMs) have been used to understand the semantic relationships within SLEEC capabilities to produce more accurate results in the analysis [ 7 ] . We believe our tool represents a promising direction for increasing the explainability and understandability of requirement elicitation inconsistencies for non-technical users. In the future, we plan to extend our work to support the verification of systems against SLEEC requirements. Appendix A Description of the Demonstration We present the demonstration in