OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering
Mia Mohammad Imran, Tarannum Shaila Zaman · 2025
Read OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering by Mia Mohammad Imran, Tarannum Shaila Zaman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15979.txt.
Introduction
Mia Mohammad Imran and Tarannum Shaila Zaman present OLAF, a framework for language model based labeling in software studies, in a 2025 position paper. Researchers increasingly use language models to label commits, issues, and descriptive artifacts. The steadiness of those labels and whether they can be repeated, though, are rarely measured with standard tools.
The authors argue annotation with language models should be treated as a measuring process, not mere automation. OLAF organizes the key ideas: steadiness, calibration, drift, consensus, combining, and openness. Many studies omit setup details that decide if work can be repeated, the paper notes. It aims to spur methods debate and future hands on work toward open labeling practice.
Catalog record
| CREATOR | Mia Mohammad Imran, Tarannum Shaila Zaman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15979
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering Mia Mohammad Imran, Tarannum Shaila Zaman Abstract . Large Language Models (LLMs) are increasingly used in empirical software engineering (ESE) to automate or assist annotation tasks such as labeling commits, issues, and qualitative artifacts. Yet the reliability and reproducibility of such annotations remain underexplored. Existing studies often lack standardized measures for reliability, calibration, and drift, and frequently omit essential configuration details. We argue that LLM-based annotation should be treated as a measurement process rather than a purely automated activity. In this position paper, we outline the Operationalization for LLM-based Annotation Framework (OLAF) , a conceptual framework that organizes key constructs: reliability, calibration, drift, consensus, aggregation , and transparency . The paper aims to motivate methodological discussion and future empirical work toward more transparent and reproducible LLM-based annotation in software engineering research. 1. Introduction Empirical software engineering depends on labeled data, such as issues identified as bugs, code comments marked as self-admitted technical debt, or developer commit messages classified by intent ( 14 ) . Traditionally, producing such annotations has required substantial human effort. Manual annotation, though considered as gold standard, is labor-intensive and can be inconsistent ( 17 ) . The rise of large language models (LLMs) has transformed this process. In recent years, Researchers have employed models such as GPT, Claude, or Llama to generate preliminary labels or even to replace manual annotation entirely ( 13 ; 10 ; 2 ) . While this approach accelerates data production, it also introduces new methodological concerns. When an LLM assigns a label, it effectively functions as a measurement instrument . Just as code metrics must be validated before use, labels produced by LLMs need to be tested for reliability and reproducibility. As noted by Wagner et al., many SE studies involving LLMs neglect to report standard measures of reliability or calibration ( 25 ) . Baumann et al. observe that minor configuration choices, such as prompt phrasing or sampling temperature, can completely reverse outcomes, a form of prompt drift – which they refer as LLM hacking ( 6 ) . Ahmed et al. find that LLMs can perform on par with human annotators, but only under carefully controlled evaluation conditions ( 2 ) . Meanwhile, Wang et al. empirically studied LLM-as-a-Judge in SE tasks and noted that LLMs could potentially substitute human evaluations in certain SE tasks while they may be provide satisfactory and consistent performance in other tasks ( 26 ) . These findings highlight a key gap: the lack of operationalization—the formal link between abstract concepts (e.g., “label accuracy”) and measurable procedures ( 15 ) . Without clear operational definitions for constructs like reliability, consensus, or drift, ESE cannot guarantee reproducibility. While existing guidelines, such as those by ACM and Baltes et al. ( 4 ) , offer general advice on adopting LLMs in SE, they do not specifically address the need for operationalizing annotations in SE tasks. To fill this gap, we introduce the Operationalization for LLM-based Annotation Framework (OLAF) , a structured approach that treats LLM-based annotation as a measurable and auditable process. To lay the groundwork for OLAF, we first explore why operationalization is essential and how current practices fall short in ensuring methodological rigor. 2. Need for Operationalization in LLM-based Annotation 2.1. Fragmented Annotation Practices The challenge of using LLMs for annotation lies not in performance but in methodological rigor. Many studies lack clear label definitions and consistent operationalization of reliability, calibration, or consensus, and often omit key configuration details (e.g., temperature). Based on recent evaluations ( 25 ; 6 ; 2 ; 26 ) , we summarizes these weaknesses and their implications. 2.1.1. Missing Details Ahmed et al. ( 2 ) noted that even when models reach human-level accuracy, details such as prompt templates, model versions, and decoding parameters are sometimes omitted. Without these details, replication becomes uncertain. Several patterns appear from recent observations ( 2 ; 6 ; 25 ) : • Limited reliability reporting. Some studies provide only raw accuracy without chance-corrected measures (e.g., Cohen’s \kappa ). • Incomplete configuration description. Prompt templates, model versions, and decoding parameters are often underspecified. • Minimal calibration assessment. Model confidence and stability across versions are rarely examined. For example, researchers reported the performance difference between different GPT model versions during annotation ( 3 ) . 2.1.2. Configuration Sensitivity Baumann et al. ( 6 ) found that minor configuration changes, such as prompt wording, temperature, or model version, can alter results in text-annotation tasks. This sensitivity complicates the expectation of stable measurement conditions. In practice, it appears as: • Prompt variation: minor lexical variations (e.g., “bug fix” vs. “defect repair”) yield different outputs. The problem of Prompt Perturbation ( 8 ) is still an under-explored area in general; • Model variation: API updates modify model behavior under the same identifier; • Sampling variation: stochastic decoding introduces variability across runs. 2.1.3. Unstandardized Constructs and Metrics ESE commonly employs metrics for inter-rater reliability and measurement validity. In contrast, LLM-based studies rarely define comparable constructs for annotation quality. Ahmed et al. ( 2 ) propose human-model agreement ( \kappa_{H\text{-}M} ) and model-model consensus ( \rho_{M\text{-}M} ) as candidate indicators, while Baumann et al. discuss model sensitivity to configuration ( 6 ) . 2.1.4. Consequences for Research Validity The absence of operationalization undermines the validity of results across all levels of empirical reasoning. Construct validity suffers when labels lack defined meaning; internal validity is threatened by uncontrolled configuration sensitivity; and external validity is compromised by opaque or evolving model configurations that hinder replication. 2.2. Problem Summary LLM-based annotation has advanced rapidly, but its methodological foundations remain underdeveloped. Existing studies follow fragmented practices, report reliability inconsistently, and rely on loosely defined constructs, making results difficult to interpret or reproduce. As a result, LLM-generated labels are often treated as measurements without being properly validated. Recent methodological analyses ( 2 ; 6 ; 25 ) highlight the need to reconceptualize annotation as a structured measurement process rather than a purely automated task. Moreover, when annotation targets lack a clear gold standard or involve irreducible expert disagreement, reliability and calibration become ill-defined, rendering probabilistic outputs unsuitable for safety-critical or compliance-sensitive settings. Addressing these challenges requires standardized constructs, transparent configuration reporting, and reproducible aggregation methods. In the following section, we introduce OLAF , a framework designed to meet these requirements. 3. Operationalization for LLM-Based Annotation Framework (OLAF) We provide detailed description of the proposed framework below. 3.1. Measurement of LLM Annotation The methodological issues that we discussed earlier, fragmented practices, configuration sensitivity, and undefined constructs, stem from treating LLM annotation as an automation task rather than as measurement . When a model assigns a label, it acts as a measurement instrument whose reliability must be empirically established. Table 1. Operationalization for LLM-based Annotation Framework (OLAF) Construct Measures Metric / Indicator Reporting Focus Annotator’s Reliability ( 11 ) Annotator agreement Cohen’s \kappa ( 11 ) , Krippendorff’s \alpha ( 18 ) Stability of individual annotators Consensus ( 7 ) Group agreement Correlation metrics Collective alignment Aggregation ( 19 ) Combining annotators i) probabilistic ( 16 ) : Dawid-Skene, GLAD, MACE; ii) Majority voting ( 24 ) Quality of fused labels Transparency ( 20 ) Documentation quality