Why should more than two reliability coefficients be reported in triage studies? Authors: Amir Mirhaghi Published: 2026 ABSTRACT Abstract“This submission is a Matters Arising article type; therefore, no abstract is required. Please refer to the submission guidelines of the journal”.Keywords: Krippendorff’s alpha, Kappa, Reliability, Triage Dear Editor, We read with great interest the manuscript entitled “An inter-rater reliability study of a modified version of SATS as a prehospital triage tool” by Deshayes et al. [1]. The authors investigated the inter-rater reliability of a modified version of the South African Triage Scale (SATS) when applied by emergency medical services (EMS) personnel in a Swedish region. They reported poor agreement for overall triage classification, with a Krippendorff’s alpha of 0.602 (95% CI: 0.423–0.712), which changed only marginally after excluding manual up-triage (Krippendorff’s α = 0.621; 95% CI: 0.442–0.755). Furthermore, when triage categories were dichotomized into high- and low-acuity groups, inter-rater reliability decreased to a Krippendorff’s alpha of 0.47 (95% CI: 0.33–0.60). Several methodological issues warrant further discussion. First, the authors correctly acknowledge that Krippendorff’s alpha is generally more conservative than kappa [1]. More precisely, Krippendorff’s alpha is considered one of the most conservative measures of inter-rater reliability, making comparisons with studies reporting alternative reliability coefficients challenging. Esser and Aksu demonstrated this discrepancy by evaluating the same ordinal dataset using several reliability statistics [2]. While Kendall’s coefficient of concordance (W = 0.69), Pearson’s correlation coefficient (r = 0.68), and the intraclass correlation coefficient (ICC = 0.93) indicated moderate to excellent agreement, Krippendorff’s alpha was considerably lower (α = 0.31) [2]. These findings suggest that studies employing Krippendorff’s alpha should also report additional reliability measures to facilitate interpretation and comparison with the existing literature. In the present study, the ICC or Fleiss’ kappa would have been appropriate complementary statistics for assessing agreement among multiple raters. Nevertheless, evidence regarding the performance of Krippendorff’s alpha relative to other reliability coefficients remains inconsistent. Zapf et al. demonstrated that Krippendorff’s alpha and Fleiss’ kappa yield comparable estimates when applied to nominal data without missing values, concluding that either statistic can be recommended for assessing inter-rater reliability under such conditions [3]. It is also noteworthy that unweighted kappa and linear weighted kappa generally produce lower estimates of agreement than quadratic weighted kappa or the ICC. For example, de Raadt et al. reported unweighted and weighted kappa values of 0.50 and 0.78, respectively, for the same dataset [4]. Therefore, unweighted kappa may produce values more comparable to Krippendorff’s alpha than other reliability coefficients, although further methodological studies are required to confirm this observation. The authors also demonstrated that discrepancies among reliability coefficients become more pronounced as the level of agreement increases, particularly between quadratic weighted kappa and correlation-based measures such as the ICC and Pearson’s correlation coefficient. Conversely, when agreement is only moderate, the various reliability coefficients tend to converge. Second, unlike kappa statistics, reducing the number of categories appeared to decrease inter-rater reliability when assessed using Krippendorff’s alpha. Specifically, dichotomizing triage levels into high- and low-acuity categories reduced Krippendorff’s alpha from 0.60 to 0.47 in the current study. This finding contrasts with previous reports using kappa statistics. Jakobsson et al. demonstrated that collapsing four response categories into two increased the prevalence within each category and consequently increased kappa from 0.709 to 0.857 [5]. Comparing to kappa, ICC is generally less sensitive to changes in category prevalence. The observed behavior of Krippendorff’s alpha deserves further explanation. Clarifying why Krippendorff’s alpha responded differently from kappa in this analysis would improve readers’ understanding of the reported findings. Furthermore, presenting the raw contingency table of inter-rater classifications would enable readers to calculate alternative reliability coefficients and independently compare the results. Third, systematic reviews have consistently shown that weighted kappa is the most frequently reported statistic for assessing inter-rater reliability in triage studies [6]. Consequently, the exclusive use of Krippendorff’s alpha complicates direct comparisons with previous studies and may limit the inclusion of these findings in future evidence syntheses or meta-analyses. Kappa statistics remain among the most informative measures of agreement because they account for agreement expected by chance, unlike simple percentage agreement or correlation coefficients [5, 7]. However, unweighted kappa assigns equal weight to all disagreements regardless of their magnitude, making it less appropriate for ordinal scales. Weighted kappa was developed to address this limitation by assigning greater penalties to larger disagreements. It should also be noted that weighted kappa is mathematically equivalent to the ICC under certain assumptions and, like the ICC, may overestimate agreement. Although kappa statistics are influenced by prevalence and observer bias, these limitations can be partially addressed by calculating the prevalence-adjusted and bias-adjusted kappa (PABAK) [8]. In conclusion, we commend the authors for applying Krippendorff’s alpha to evaluate the reliability of a prehospital triage system. Nevertheless, reporting additional reliability coefficients, together with the raw inter-rater agreement table, would substantially improve the interpretability, comparability, and generalizability of the findings. Such information would also facilitate comparison with the existing body of triage reliability research, in which kappa remains the most commonly reported measure of agreement.