Bug Classification in Quantum Software: A Rule-Based Framework and Its Evaluation

Mir Mohammad Yousuf, Shabir Ahmad Sofi · 2025

Read Bug Classification in Quantum Software: A Rule-Based Framework and Its Evaluation by Mir Mohammad Yousuf, Shabir Ahmad Sofi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-10397.txt.

Introduction

Mir Yousuf and Shabir Sofi present a rule-based system to sort bugs in quantum software in this 2025 paper. Sorting bug reports by type and severity helps teams fix the right problems first. Quantum software adds its own bug kinds, tied to circuits and gates, that generic tools miss.

The system uses keyword rules tuned to quantum terms. To test it, the authors hand-sorted a sample of 4,984 issues from 12,910 across 36 project archives. The system reached up to 85 percent accuracy, with strong agreement on bug type and category, but weak agreement on severity. Classic bugs made up about two thirds of issues, and quantum-specific bugs about a quarter. Most issues were low severity.

Catalog record

CREATORMir Mohammad Yousuf, Shabir Ahmad Sofi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-10397

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bug Classification in Quantum Software: A Rule-Based Framework and Its Evaluation Mir Mohammad Yousuf, Shabir Ahmad Sofi Abstract Accurate classification of software bugs is essential for improving software quality. This paper presents a rule-based automated framework for classifying issues in quantum software repositories by bug type, category, severity, and impacted quality attributes, with additional focus on quantum-specific bug types. The framework applies keyword and heuristic-based techniques tailored to quantum computing. To assess its reliability, we manually classified a stratified sample of 4,984 issues from a dataset of 12,910 issues across 36 Qiskit repositories. Automated classifications were compared with ground truth using accuracy, precision, recall, and F1-score. The framework achieved up to 85.21% accuracy, with F1-scores ranging from 0.7075 (severity) to 0.8393 (quality attribute). Statistical validation via paired t-tests and Cohen’s Kappa showed substantial to almost perfect agreement for bug type (k = 0.696), category (k = 0.826), quality attribute (k = 0.818), and quantum-specific bug type (k = 0.712). Severity classification showed slight agreement (k = 0.162), suggesting room for improvement. Large-scale analysis revealed that classical bugs dominate (67.2%), with quantum-specific bugs at 27.3%. Frequent bug categories included compatibility, functional, and quantum-specific defects, while usability, maintainability, and interoperability were the most impacted quality attributes. Most issues (93.7%) were low severity; only 4.3% were critical. A detailed review of 1,550 quantum-specific bugs showed that over half involved quantum circuit-level problems, followed by gate errors and hardware-related issues. Keywords Quantum software, Quantum software bugs, rule based classification,manual classification, Qiskit, quantum software testing 1 Introduction Quantum software engineering presents unique challenges that stem from the hybrid nature of quantum programs, which combine classical orchestration with quantum circuit execution. These challenges include managing the complexity of quantum operations, debugging under hardware noise, and dealing with limited observability in quantum states de2022software ; gay2005communicating . As the quantum software ecosystem matures, understanding and managing software defects becomes essential to ensure reliability, maintainability, and performance in quantum applications. Automated bug classification plays a vital role in software quality assurance by enabling systematic defect analysis, supporting large-scale empirical studies, and informing tooling and development practices yang2022comprehensive ; jonsson2016automated . In classical software domains, both rule-based and machine learning-based classifiers have been employed to categorize issues by type, severity, and quality impact with considerable success yang2022survey ; ahmed2021capbug ; bhandari2023buggin . However, there is a significant gap in tools and techniques specifically designed to classify bugs in quantum software, where the nature and semantics of defects often differ markedly from classical counterparts haghparast2023quantum ; li2021understanding ; de2022software ; zhao2020quantum . In this study, we introduce a rule-based automated classification framework tailored to the quantum software domain. The framework classifies issues across five dimensions: 1. Bug type (quantum or classical). 2. Bug category (e.g., logical, syntax, quantum-specific). 3. Severity (critical, high, medium, low). 4. Impacted quality attribute (e.g., maintainability, reliability, performance). 5. Type of quantum-specific issue (e.g., quantum gate errors, measurement errors, transpilation issues). This system uses curated keyword lists and heuristic rules grounded in domain knowledge of quantum computing and common failure modes in quantum programs. We collected 12,910 issues from the 36 repositories of Qiskit and performed the classification. To evaluate its effectiveness, we conducted a large-scale manual annotation of 4,984 GitHub issues from 36 repositories within the Qiskit ecosystem. Issues were selected using random stratified sampling to ensure representation across severity levels and bug types within this specific framework, but not across different categories of quantum software. Each issue was manually labeled for bug type, category, severity, and impacted quality attribute, forming a ground truth dataset for benchmarking. We assessed the framework’s performance against this ground truth using standard metrics including accuracy, precision, recall, and F1-score, as well as inter-rater agreement via Cohen’s Kappa. We also employed paired t-tests to compare manual and automated classifications statistically. Our findings demonstrate that the framework achieves substantial to near-perfect agreement ( \kappa>0.69 ) for bug type, category, and quality attribute classification. However, severity classification showed only slight agreement ( \kappa=0.162 ), suggesting it is more difficult to infer severity heuristically. These results highlight the framework’s utility for large-scale issue analysis in quantum software while also identifying areas for future enhancement. This work offers a reproducible and interpretable approach to quantum bug classification and supports broader efforts to build robust quantum software analysis tools. 1.1 Contributions This work makes the following key contributions toward improving automated bug classification in quantum software: 1. A Rule-Based Framework for Multi-Dimensional Bug Classification: We propose a rule-based classification framework that automatically labels quantum software issues across five dimensions: bug type, bug category, severity level, impacted quality attribute, and quantum-specific subtype. The framework is designed to be interpretable, adaptable, and grounded in domain-specific keyword analysis. 2. A Manually Annotated Dataset of Quantum Software Issues: We construct a manually labeled dataset comprising 4984 issues from 36 Qiskit repositories, annotated along all five classification axes. This dataset serves as a benchmark for evaluating bug classification techniques in the quantum domain. 3. Empirical Evaluation of Rule-Based Classification: We evaluate the classification performance of our rule-based system using standard metrics (precision, recall, F1-score) against the annotated dataset. Additionally, we apply a paired t-test to assess the statistical significance of performance differences and use Cohen’s kappa to measure inter-rater agreement between the rule-based predictions and the human annotations. The results demonstrate that rule-based approaches can deliver high accuracy, strong agreement with human annotations, and interpretability in domains with limited labeled data and strong domain-specific characteristics. 1.2 Overall structure of the paper The paper is structured to provide a comprehensive exploration of a rule-based framework for automated classification of quantum software issues. Section 1 1 introduces the motivation and scope of the study, followed by a summary of the key contributions of the work. Section 2 reviews the related work, positioning the paper within the broader context of research on software bug classification and quantum software engineering. Section 3 presents the methodology, beginning with the process of dataset construction and manual annotation. It then outlines the multiple classification dimensions considered in this study, including classical vs. quantum classification, mapping of issues to bug categories, quality attributes, severity levels, and quantum-specific bug categories, while emphasizing transparency and reproducibility of the process. Section 4 reports the results of the evaluation. It includes a comparison of manual and automated classifications through statistical agreement, and presents classification performance metrics. The section also explores the distribution of bug types, bug categories, quality attributes, severity levels, and quantum-specific bug categories, along with a summary of the key findings. Section 5 offers the conclusion, summarizing the primary insights and outcomes of the study. Section 6 discusses directions for future work to extend and refine the framework. Finally, Section 7 addresses threats to validity, covering internal, external, and construct validity to assess the robustness and generalizability of the study’s findings. 2 Related Work Bug classification has been a longstanding area of interest in software engineering, with approaches ranging from manual taxonomies to automated machine learning models. Early work focused on categorizing bugs based on source code analysis, defect patterns, and fault types pan2009toward ; kim2008classifying . More recent studies have employed natural language processing (NLP) techniques to classify bugs using issue descriptions and commit messages dos2020commit ; zhou2017automated ;

Read the complete text