When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

Anna Sofia Lippolis, Mohammad Javad Saeedizade, Eva Blomqvist, Andrea Giovanni Nuzzolese, Aldo Gangemi, Robin Keskisärkkä · 2026

Read When CQs Go Wrong: Challenges in CQ Verification with OE-Assist by Anna Sofia Lippolis, Mohammad Javad Saeedizade, Eva Blomqvist, Andrea Giovanni Nuzzolese, Aldo Gangemi, Robin Keskisärkkä (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24619.txt.

Introduction

Lippolis and colleagues study what makes competency questions hard to verify against an ontology. Such questions state, in natural language, what an ontology, a formal model of a domain, must be able to answer. CQ checking tests the model against them. The process is slow and error prone. Linguistic nuance must align precisely with formal constructs, and ambiguous or overcomplex questions poison the decisions downstream.

Nineteen people performed checking on twenty tasks inside OE-Assist. That is an engineering environment, with a language model assistant supporting evaluation. Analysing where they struggled identifies what makes a question challenging, and which supports improve performance. The headline result is about process. A tool to refine questions before they are published is necessary, scrubbing ambiguity before later engineering phases inherit them. Lippolis's 2026 study treats question quality as groundwork. Checking fails upstream, at the writing desk, more often than at the model.

Catalog record

CREATORAnna Sofia Lippolis, Mohammad Javad Saeedizade, Eva Blomqvist, Andrea Giovanni Nuzzolese, Aldo Gangemi, Robin Keskisärkkä
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24619

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When CQs Go Wrong: Challenges in CQ Verification with OE-Assist Anna Sofia Lippolis, Mohammad Javad Saeedizade, Eva Blomqvist, Andrea Giovanni Nuzzolese, Aldo Gangemi, Robin Keskisärkkä Abstract Competency Questions (CQs) are the central component of CQ-verification, an established process in which an ontology is evaluated against a set of natural language questions to determine whether the intended purpose of the ontology has been properly modelled. However, CQ-verification is often time-consuming and error-prone, as it requires careful interpretation of linguistic nuances and precise alignment with formal ontology constructs. Ambiguities and complexity in CQs can further complicate this process, leading to inconsistent modelling decisions and verification outcomes. In this paper, we investigate what makes a CQ challenging and possible solutions to enhance the users’ performance in the CQ-verification process. We experimented with the data of 19 participants who performed CQ-verification on 20 tasks using an LLM assistant to support ontology evaluation. The results show the necessity of a tool to refine CQs before publishing them to avoid ambiguity or excessive complexity in later phases of the ontology engineering process. * * footnotetext: Equal contribution. 1 Introduction and related work Competency Questions (CQs), and in particular, verification CQs [ 4 ] , are fundamental tools in knowledge engineering. They are requirements that an ontology must be able to answer. In fact, the long-standing practice of CQ-verification [ 3 ] evaluates a produced ontology based on whether it correctly models the knowledge necessary to answer a predefined set of CQs. Although conceptually straightforward, this process is time-consuming and prone to interpretation errors, which can derive from the set of CQs itself. Indeed, as noted in many studies, quality dimensions such as ambiguity, readability, and complexity in CQs [ 1 , 2 ] could result in inconsistent modelling decisions and verification outcomes, and even carefully curated CQ benchmarks are not immune to such issues. Still, very little guidance exists for formulating CQs and assessing whether they constitute good CQs [ 4 , 5 ] . At the same time, the increasing capabilities of LLMs have introduced new possibilities for automating and assisting CQ-related tasks from a variety of scenarios, including CQ generation, refinement, formalisation, and ontology generation and evaluation [ 6 , 8 , 10 , 9 ] . As ontology development becomes more automated, understanding how ambiguity in CQs affects ontology development and verification performance becomes increasingly important. In this work, we empirically investigate ambiguity and complexity in CQs within the context of ontology evaluation. We conducted a controlled CQ-verification experiment with 19 ontology engineers in OE-Assist [ 7 ] , whose resources are available on Github , and we analysed free-text user comments from that experiment to study more variables affecting users’ performance. Our main research question is “How can CQ formulations be evaluated to identify the features that make them challenging in CQ-verification?” By examining users’ feedback and the time spent on each CQ w.r.t the length of the ontology, CQ complexity, and the time they spent on the experiment, we aim to provide empirical insights that inform ontology engineering practices to formulate CQs. The final aim of this work is to explore pitfalls that occur during the CQ creation process and to investigate how such issues might be identified, potentially informing future tool support. 2 Experimental setup This section summarises the semi-automatic, human-in-the-loop component of the OE-Assist study and then outlines the additional factors not examined in the original work, together with the empirical approach adopted here. Lippolis et al. [ 7 ] presented OE-Assist, a prototype designed to support in a semi-automatic way the CQ-verification workflow of Blomqvist et al. [ 3 ] . In CQ-verification, ontology engineers determine whether an ontology captures the information required to answer a natural-language CQ. Operationally, a CQ is considered modelled if a SPARQL test query can be built that returns the expected answer; otherwise it is not. The key elements of the OE-Assist experiment are as follows. Participants: N=19 ontology engineers from academic networks with varying levels of expertise. Conditions: Each participant completed verification tasks under two settings: assisted (LLM-generated suggestions available) and unassisted (Protégé only). Each participant indicated whether a CQs was modelled or not and the level of difficulty on a Likert scale from 1 to 5. We also recorded qualitative feedback in free text. Outcome of OE-Assist: The results indicate strong reliance on LLM suggestions: correct suggestions improved performance, whereas incorrect or misleading suggestions reduced performance relative to the unassisted condition. Here, we investigate how characteristics of CQ formulation and user stories relate to human verification performance. Our objectives are (1) to identify CQ- and ontology-level factors associated with decision accuracy, perceived difficulty and response time, and (2) to derive actionable recommendations for a future tool that serves as a CQ pitfall scanner. Concretely, we analyse the following variables extracted from the OE-Assist session notes: User correctness: Denoting whether the participant’s CQ-verification decision (“CQ modelled” vs. “not modelled”) matches the ground truth. Perceived Difficulty: A score from 1 (=not difficult at all)-5 (=very difficult) given by users for a CQ-verification task. Decision duration: Time the participant spent on a given CQ before submitting a decision. Ontology size: An ordinal descriptor of the OWL artefacts used in the study (“Small”, “Medium”, “Large”) derived from axiom count. CQ complexity: A readability score computed from the CQ text. Following [ 2 ] , we compute Flesch–Kincaid Grade Level and the Gunning Fog Index and treat these indices as continuous predictors of CQ complexity. Our analysis proceeds in two phases. We ran pairwise correlation tests (Spearman’s ρ and Kendall’s τ ) and appropriate pairwise comparisons to identify predictors of user correctness. Second, we perform a qualitative analysis of the session notes and feedback from the users. 3 Results Figure 1: Correlation analysis of CQ completion time, user correctness, ontology size, and the two complexity metrics introduced by Alharabi et al. [ 2 ] . The figure additionally includes a box plot of completion time stratified by ontology size and statistics of time spent on each CQ duration and CQ complexity. The results indicate that only one statistically significant association was observed. Specifically, a moderate positive correlation was found between decision duration and perceived difficulty (Spearman’s \rho=0.42 ; Kendall’s \tau=0.34 ; both p<0.001 ), suggesting that tasks requiring longer completion times were more likely to be rated as difficult by users. No statistically significant associations were identified among the other examined variable pairs, including decision duration and CQ complexity, decision duration and ontology size, or CQ complexity and ontology size. These findings suggest that, within the present dataset, perceived difficulty is primarily related to time spent on the task. Table 1: Semantic range of the word “resource” and some European equivalents. Language Nearest word Expanded meaning in English English resource supply; asset; means; source; natural resource French ressource resource; supply; financial means ; personal capability German Ressource resource; raw material; energy supply; economic asset Spanish recurso resource; means; financial means ; legal appeal ; rhetorical device Portuguese recurso resource; means; financial means ; legal appeal Italian risorsa resource; asset; personnel; financial means Swedish resurs resource; asset; funding; personnel allocation; However, qualitative feedback collected during the experiment revealed three major issues that affected participants’ perceived difficulty and

Read the complete text