Position: Early-Stage Quality Assurance in Annotation Pipelines Is More Cost-Effective Than Late-Stage Validation
Sunil Kothari, Sumukha Sharma Thoppanahalli Chandramouli, Naman Khandelwal, Parth Kulshreshtha, Ashi Jain, Kriti Banka, Tanuja Chintada, Venkata Triveni, Gulipalli Praveen Kumar, Manish Mehta, Tao Liu · 2026
Read Position: Early-Stage Quality Assurance in Annotation Pipelines Is More Cost-Effective Than Late-Stage Validation by Sunil Kothari, Sumukha Sharma Thoppanahalli Chandramouli, Naman Khandelwal, Parth Kulshreshtha, Ashi Jain, Kriti Banka, Tanuja Chintada, Venkata Triveni, Gulipalli Praveen Kumar, Manish Mehta, Tao Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15714.txt.
Introduction
Sunil Kothari and ten colleagues press for early checks in annotation pipelines over late validation. Data quality caps model gains, yet work fixates on methods, not timing. Software lore shows late defects cost 4 to 100 times more. The team sorts triggers into pre-annotation, post-annotation, and post-review. An error-propagation model pins down when timing shifts rates versus only costs. A survey of 47 papers finds only 4 percent report validation timing. The team urges reporting QA timing and running timing experiments.
Catalog record
| CREATOR | Sunil Kothari, Sumukha Sharma Thoppanahalli Chandramouli, Naman Khandelwal, Parth Kulshreshtha, Ashi Jain, Kriti Banka, Tanuja Chintada, Venkata Triveni, Gulipalli Praveen Kumar, Manish Mehta, Tao Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31721 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15714
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position: Early-Stage Quality Assurance in Annotation Pipelines Is More Cost-Effective Than Late-Stage Validation Sunil Kothari, Sumukha Sharma Thoppanahalli Chandramouli, Naman Khandelwal, Parth Kulshreshtha, Ashi Jain, Kriti Banka, Tanuja Chintada, Venkata Triveni, Gulipalli Praveen Kumar, Manish Mehta, Tao Liu Abstract This position paper argues that the machine learning community should prioritize early-stage quality assurance in annotation pipelines over the prevailing practice of late-stage validation. Data quality bottlenecks increasingly limit foundation model improvement, yet quality assurance research focuses almost exclusively on validation methods rather than validation timing. When validation occurs—not merely what validation methods are employed—fundamentally determines both error rates and annotation costs. This temporal neglect is puzzling given the well-established “shift-left” principle from software engineering, where empirical studies demonstrate 4–100 × \times cost multipliers for defects detected in later development stages ( Boehm 1981 ; Shull et al. 2002 ) . Annotation pipelines, we argue, exhibit analogous dynamics: errors caught before annotation begins cost a fraction of those discovered after review cycles complete. We propose a taxonomy of three QA trigger points —pre-annotation ( T 0 T_{0} ), post-annotation ( T 1 T_{1} ), and post-review ( T 2 T_{2} )—that decompose annotation workflows into discrete validation opportunities. A parametric error-propagation model formalizes when timing affects final error rates (when stage-specific detection rates differ) versus only economics (when they are equal), making timing a measurable design variable rather than a configuration afterthought. A survey of 47 recent papers reveals that only 4% report when validation occurs, a striking gap given timing’s demonstrated impact in adjacent fields. Without explicit attention to QA timing, the community risks optimizing validation methods while ignoring the structural variable that may matter most. Acting on this position requires three steps: researchers should report QA timing configurations alongside validation methods; annotation platforms should expose timing as a first-class parameter; and the community should run controlled experiments that measure stage-specific detection rates directly, operationalizing the shift-left principle for annotation pipelines. † † affiliationnotice: Equal contribution 1 Introduction We argue that quality assurance timing—specifically, whether validation occurs before annotation, after annotation, or after review—is a critical design variable that the machine learning community has systematically overlooked, and that early-stage QA is more cost-effective than late-stage validation for most annotation workflows. The data-centric AI paradigm has established that training data quality fundamentally constrains model performance ( Ng 2021 ; Sambasivan et al. 2021 ; Whang et al. 2023 ) . Recent analyses have revealed pervasive label errors across major benchmarks: Northcutt et al. 2021b identified error rates averaging 3.3% in ImageNet, CIFAR, and other widely-used test sets, while Beyer et al. 2020 found that 6% of ImageNet validation labels require correction. For video annotation pipelines processing thousands of hours of footage at frame rates of 16–30 fps, even small per-frame error rates compound into substantial quality degradation ( Dave et al. 2020 ; Voigtlaender et al. 2019 ) . The research community has responded with increasingly sophisticated validation methods. Confident learning enables statistical detection of label errors without ground truth ( Northcutt et al. 2021a ) . Vision-language models such as GPT-4V and Qwen2-VL provide semantic verification capabilities ( Wang et al. 2024 ; OpenAI 2023 ) . Multi-annotator consensus methods model annotator reliability and aggregate labels accordingly ( Dawid & Skene 1979 ; Raykar et al. 2010 ; Goh et al. 2022 ) . Inter-annotator agreement metrics quantify annotation consistency ( Artstein & Poesio 2008 ; Cohen 1960 ; Krippendorff 2011 ) . These advances focus exclusively on what to validate and how to detect errors. Yet they share a critical blind spot: the question of when validation should occur in the annotation pipeline receives virtually no systematic attention. We surveyed 47 articles on CVPR, NeurIPS and ICML annotation quality (2022–2024) and found that only 2 articles (4%) explicitly reported the stage of the pipeline at which validation was applied. This omission matters because the same validation method produces different outcomes depending on when it executes. Consider a video annotation pipeline in which machine learning pre-annotation produces bounding boxes with a 15% error rate. If quality assurance runs only after human review (late-stage), these errors propagate throughout the pipeline before detection, consuming annotator time on corrections that could have been prevented. If quality assurance runs before human annotation begins (early-stage), errors are caught before human effort is invested, potentially reducing both error rates and costs. Software engineering learned this lesson decades ago. The “shift-left” principle, formalized by Boehm 1981 and validated in subsequent empirical studies ( Shull et al. 2002 ; Jones 2008 ; McConnell 2004 ) , establishes that defects caught in early development stages cost substantially less to remediate than those caught later. Manufacturing quality management and ML technical debt research encode similar intuitions, which we examine in Section 3 . We hypothesize that annotation pipelines exhibit dynamics analogous to software development and manufacturing: errors detected early cost less to remediate than errors detected late. This paper does not prove this hypothesis empirically—such proof requires controlled experiments comparing identical validators across pipeline stages, which we have not conducted. Rather, we argue that the hypothesis is sufficiently plausible, and the current neglect of timing sufficiently systematic, that the research community should treat QA timing as a first-class research question. The remainder of this paper proceeds as follows. Section 2 introduces our taxonomy of QA trigger points and situates it within annotation workflow structure. Section 3 presents evidence that timing is currently invisible in both research and practice. Section 4 develops a parametric model clarifying when timing affects error rates versus only economics. Section 5 presents the full configuration taxonomy. Section 6 engages substantively with alternative views. Section 7 presents our call to action, Section 8 describes the limitations of this approach and Section 9 concludes. 2 QA Trigger Points: A Framework Before presenting our argument, we must establish precise terminology for discussing when quality assurance occurs in annotation pipelines. We introduce the concept of QA trigger points —discrete moments in the annotation workflow where validation can be invoked—and define three canonical trigger points that capture the structure of typical annotation pipelines. 2.1 Annotation Pipeline Structure Modern annotation pipelines for computer vision tasks typically proceed through three phases ( Roh et al. 2019 ; Monarch 2021 ) . First, machine learning pre-annotation generates initial predictions (bounding boxes, segmentation masks, tracking identities) that serve as starting points for human annotators ( Yao et al. 2012 ; Papadopoulos et al. 2017 ) . Second, human annotation refines, corrects, or validates these predictions, with annotators adding missing objects, adjusting boundaries, and correcting labels ( Su et al. 2012 ; Kovashka et al. 2016 ) . Third, human review provides quality control, with reviewers accepting, rejecting, or requesting revisions to submitted annotations ( Daniel et al. 2018 ; Vaughan 2017 ) . This three-phase structure creates natural boundaries where quality assurance can intervene. We formalize these as trigger points. 2.2 Trigger Point Definitions ML Pre-annot. T_{0} Annotation T_{1} Review T_{2} Output Pre-annot. QA Post-annot. QA Post-review QA Figure 1 : QA trigger points in annotation pipelines. T_{0} occurs after ML pre-annotation but before human work. T_{1} occurs after annotation but before review. T_{2} occurs after review. Each trigger point enables different validation capabilities and