Reducing Maintenance Burden in Behaviour-Driven Development: A Paraphrase-Robust Duplicate-Step Detector with a 1.1M-Step Open Benchmark Journal: Information and Software Technology

Ali Hassaan Mughal, Noor Fatima, Muhammad Bilal · 2026

Read Reducing Maintenance Burden in Behaviour-Driven Development: A Paraphrase-Robust Duplicate-Step Detector with a 1.1M-Step Open Benchmark Journal: Information and Software Technology by Ali Hassaan Mughal, Noor Fatima, Muhammad Bilal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20462.txt.

Introduction

Mughal and two colleagues release the largest cross-organization corpus of behaviour-driven development steps, from 347 GitHub repositories, plus a detector for duplicate Gherkin steps. The corpus holds over a million steps, and the detector layers exact hashing, normalized Levenshtein distance, sentence-transformer cosine similarity, and a hybrid of the last two. Calibration uses 1,020 hand-labelled pairs under a released rubric. The exact-duplicate rate weighted by step is 80.2 percent, with nearly nine hundred thousand eliminable step occurrences across the corpus.

Catalog record

CREATORAli Hassaan Mughal, Noor Fatima, Muhammad Bilal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47161
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20462

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reducing Maintenance Burden in Behaviour-Driven Development: A Paraphrase-Robust Duplicate-Step Detector with a 1.1M-Step Open Benchmark Journal: Information and Software Technology Ali Hassaan Mughal, Noor Fatima, Muhammad Bilal Abstract Context. Behaviour-Driven Development (BDD) suites written in Gherkin accumulate step-text duplication whose maintenance cost is established in prior work. Existing detection techniques either require running the tests (Binamungu et al., 2018-2023) or are confined to a single organisation (Irshad et al., 2020-2022), leaving a gap: a purely static, paraphrase-robust, step-level detector usable on any repository, and a public benchmark on which to calibrate it. Objective. We fill the gap by building (i) the largest cross-organisational released BDD step corpus to date, (ii) a labelled pair-level calibration benchmark for duplicate detection, and (iii) a detection layer comprising four strategies along a precision/compute-cost frontier, with an explicit consolidation-savings model that maps detected clusters to ISO/IEC 25010 quality sub-characteristics. Method. The corpus comprises 347 public GitHub repositories, 23,667 parsed .feature files, and 1,113,616 Gherkin steps, discovered via the GitHub REST Search API and tagged with SPDX licence identifiers. The detector layers exact hashing, normalised Levenshtein, sentence-transformer cosine, and a hybrid Levenshtein-banded variant. Calibration uses 1,020 step pairs manually labelled by the three authors under a released rubric, with a 60-pair overlap subset for inter-annotator agreement (Fleiss’ κ = 0.84 \kappa=0.84 ). We report precision, recall, and F 1 with bootstrap 95 % confidence intervals under two protocols (the primary rubric and a score-free second-pass relabelling) and benchmark against two classical lexical baselines (SourcererCC-style token Jaccard and NiCad-style char- n n -gram TF-IDF). Results. The step-weighted exact-duplicate rate is 80.2 %; the median-repository rate is 58.6 % (Spearman ρ = 0.51 \rho=0.51 with size). The top hybrid cluster groups 20,737 occurrences across 2,245 files. The strongest honest pair-level number is near-exact at F 1 = 0.822 {}_{1}=0.822 on score-free labels; the primary-rubric semantic F 1 = 0.906 {}_{1}=0.906 is inflated by a stratification artefact disclosed openly. Lexical baselines reach primary F 1 = 0.761 {}_{1}=0.761 (Jaccard) and 0.799 0.799 (TF-IDF). The consolidation-savings model estimates that 893,357 step-text occurrences across the corpus are amenable to maintainer-action consolidation; on the median repository, 62.5 % of step lines are eliminable. Conclusion. A public, paraphrase-robust, step-level static detector closes a real gap in BDD-quality tooling, and the released benchmark gives future detector proposals a calibration target. The tool, corpus, labelled pairs, rubric, and full analysis pipeline are released under permissive licences at https://github.com/amughalbscs16/cukereuse-release . † † highlights: Largest cross-organisational BDD step corpus released: 347 repos, 1.1M Gherkin steps First public BDD-quality pair-level benchmark: 1,020 labels, Fleiss \kappa=0.84 Static, paraphrase-robust step-level duplicate detector with calibrated F 1 Score-free relabelling protocol bounds rubric-detector circularity ISO/IEC 25010 mapping links detected clusters to maintainability properties 1 Introduction Gherkin, the near-English notation used by Cucumber, Behat, SpecFlow, pytest-bdd, behave, and related Behaviour-Driven Development (BDD) frameworks ( 30 ; 47 ) , is deployed at considerable scale: our survey of public GitHub places approximately 55,000 .feature files in indexable repositories, of which this paper assembles a subset of 347 repositories (23,667 parsed feature files, 1,113,616 steps; Section 5 ). Across that corpus, more than four out of every five steps are byte-identical duplicates of another step in the same corpus; under paraphrase-robust matching the most-repeated canonical step phrasing ( the response status is 200 OK ) appears 20,737 times across 2,245 files in 43 distinct repositories. Duplication of this shape carries documented maintenance costs. Prior BDD-quality work establishes the cost independently: 4 characterise scenario-level duplication as a first-class quality defect; 32 ; 31 catalogue practitioner-reported quality concerns where duplication and paraphrase inconsistency recur; 23 ; 22 quantify step-reuse effort impact inside an industrial single-organisation BDD suite. Three prior bodies of work occupy the adjacent territory: Binamungu et al.’s dynamic execution tracing ( 4 ; 5 ; 6 ; 8 ) (requires runnable tests; infeasible for cross-repository scale); Irshad et al.’s NCD-based single-organisation static analysis ( 24 ; 23 ; 22 ) (neither neural nor step-level); and an unpublished MSc thesis ( 43 ) . To our knowledge, no prior work applies modern sentence embeddings to Cucumber steps or performs duplicate detection at step-text level across a published cross-repository corpus. Each redundant step occurrence imposes a downstream cost: a renamed step with 1,389 occurrences (the git-town And the branches cluster) requires touching 1,389 sites unless consolidation reduces it to one. Detection enables consolidation, paraphrase-aware lints, and corpus-grounded reusable step libraries; the benefits map onto ISO/IEC 25010 maintainability and reliability sub-characteristics ( 21 ) and supply BDD champions with a measured artefact to anchor refactor-budget discussions in organisations whose practitioner-reported delivery-speed/quality trade-off is documented at scale ( 44 ) . Section 7 reports the benefits as released-corpus measurements without translating them into hour-saving or cost-saving claims that the data do not support. Contributions and research questions The paper makes six contributions: (i) the largest released cross-organisational BDD step corpus to date (347 repositories, 23,667 .feature files, 1,113,616 steps; Section 5 ), released with a Gebru-style datasheet ( 16 ) and per-step Software Package Data Exchange (SPDX) licence tags; (ii) the first published BDD-quality labelled benchmark (Section 6.1 : 1,020 step pairs, three-author labelling, Fleiss’ \kappa=0.84 on a 60-pair overlap ( 15 ) ); (iii) a score-free relabelling protocol (Section 6.3 ) that bounds the circularity between score-based rubric rules and score-based detector; (iv) two classical lexical baselines on the same benchmark with bootstrap 95 % confidence intervals (CIs) (Section 6.2 : SourcererCC-style token Jaccard ( 37 ) and NiCad-style char- n -gram Term Frequency / Inverse Document Frequency, TF-IDF ( 12 ; 36 ) ); (v) a corpus-grounded consolidation-savings model (Section 7 ) mapping detected clusters to ISO/IEC 25010 ( 21 ) maintainability and reliability sub-characteristics; and (vi) the cukereuse tool itself, released as Apache-2.0 Python alongside the corpus, benchmark, rubric, and analysis pipeline. These contributions answer four research questions (RQs): RQ1 (prevalence): how prevalent is step-text duplication in cross-organisational BDD suites and does the rate depend on licence class or repository size? RQ2 (detection quality): how do four detection strategies (exact, near-exact, semantic, hybrid) and two classical lexical baselines compare on an author-labelled benchmark with bootstrap 95 % CIs? RQ3 (methodological coupling): how much of the reported F 1 reflects circularity between score-based labels and score-based detectors? RQ4 (benefits and quality implications): what maintainability and refactoring surface area is quantifiably realisable from corpus-wide application of cukereuse, mapped to ISO/IEC 25010 dimensions? Sections 5 , 6 , 6.3 , and 7 respectively answer the four. The remainder positions the work against prior art (Section 3 , Section 4 , Section 2 ), enumerates threats to validity (Section 8 ), and concludes (Section 9 ). 2 Background 2.1 Why duplicate detection in BDD matters Defect-discovery cost grows substantially across the software lifecycle ( 10 ; 9 ) , and BDD test suites occupy a distinctive position in this cost profile because they serve simultaneously as acceptance specification, living documentation, and executable test ( 47 ; 40 ) . The general clone-detection literature establishes the link between duplication, maintainability, and refactoring outcomes: 35 survey four clone-type classes, 26 and 28 report measurable differences in bug-proneness and evolution between cloned and non-cloned code, and 2 empirically link duplication-aware refactoring to specific

Read the complete text