Critical appraisal of fairness metrics for artificial intelligence-based clinical prediction models: a scoping review Authors: João Matos, Ben Van Calster, Leo Anthony Celi, Paula Dhiman, Judy Wawira Gichoya, Richard D Riley, Chris Russell, Sara Khalid, Gary S Collins Published: 2026 ABSTRACT Predictive artificial intelligence (AI) offers an opportunity to improve clinical practice and patient outcomes but risks perpetuating biases if fairness is inadequately addressed. However, the definition of fairness remains unclear. We conducted a scoping review to identify and critically appraise fairness metrics in clinical predictive AI models. We defined a fairness metric as a metric quantifying whether a model discriminates (societally) against individuals or groups defined by sensitive attributes. We searched five databases for literature published during 2014–24, screened 820 records, included 42 studies, and extracted 63 fairness metrics. The search was limited to studies published in English. These metrics, which were classified by performance dependency, model output level, and base performance metric, revealed a fragmented landscape in the field of clinical predictive AI, with inadequate clinical validation and over-reliance on threshold-dependent metrics. 19 metrics, including only one metric for clinical use, were explicitly developed for health care. Our findings highlight conceptual challenges in defining and quantifying fairness and identify gaps in uncertainty quantification, intersectionality, and real-world applicability. Therefore, future works on clinical predictive AI models should prioritise clinically meaningful metrics. Introduction Clinical prediction models (CPMs) are typically built using regression or machine learning methods, which are collectively known as predictive artificial intelligence (AI) methods.1 CPMs can be diagnostic or prognostic; diagnostic models estimate the probability that an individual currently has a condition (typically a disease), whereas prognostic models estimate the likelihood of an individual developing a clinical outcome over a specific time period.2 Predictive AI promises to improve patient outcomes and reduce health-care costs by informing clinical decision making and risk communication.3 However, despite CPMs being abundant in the biomedical literature,4 their effects on the real world remain suboptimal,5 with only a few exceptions such as FRAX6 and QRISK3.7 Several challenges hinder successful implementation of predictive AI models, including long-standing issues with respect to reporting quality8,9 and transparency. Such issues compromise model reproducibility and independent evaluation of model performance.10 Therefore, the TRIPOD guidelines were published in 2015 to provide minimum reporting recommendations for CPMs;11 in 2024, the TRIPOD guidelines were updated as TRIPOD+AI guidelines, encompassing AI methods.12 Further, design and methodological limitations such as small sample sizes and increased risk of overfitting affect robust model development,13–16 often resulting in suboptimal or unreliable model performance17 and poor generalisability to new settings.18 Algorithmic bias, arising when data or analysis biases are encoded directly or indirectly into a model during development, adds another layer of complexity to evaluating predictive AI performance.19 These biases can arise from non-representative data,20 different underlying disease distributions,21 existing health disparities,22 biases in medical devices,23,24 and other sources.13 Such biases are inherently dependent on the notion of sensitive (or protected) attributes, such as characteristics, variables, and dimensions, along which fairness can be evaluated. Such protected characteristics vary across geographical regions,25–27 and in medical contexts, some of them might reflect biological differences that legitimately influence health outcomes.28,29 Obermeyer and colleagues’ seminal work in 2019 shed light on the issue of fairness in clinical predictive AI.22 In this study,22 a model used to predict patients’ future health needs in the USA underestimated the needs of Black patients compared with those of White patients. This bias stemmed from the model’s use of health-care costs as a proxy for health needs, which did not account for systemic disparities in access to care.22 Deploying such models risks exacerbating existing health disparities,30 leading to unfairness. Defining fairness Evaluating the performance of a CPM (eg, in terms of statistical discrimination, calibration, or clinical utility; table 1) typically focuses on population-level estimates (ie, averaged across individuals), which can mask potential differential model behaviour within that population. This masking of subgroup-level variation is also referred to as hidden stratification.37,38 Model performance, however, is expected to vary across subgroups. It is therefore important to understand the nature and magnitude of any differential model behaviour during model evaluation and identify signals suggesting unfairness. A fundamental question that arises when evaluating potentially unfair CPMs is how to define fairness? In the context of predictive AI, fairness has been conceptualised in various ways, including group fairness,42 fairness through unawareness43 or awareness,39 counterfactual fairness,35 and minimax fairness,44 among others. Despite the abundance of fairness definitions,45 there remains a notable gap in a precise and widely accepted definition of a fairness metric. Motivation and aims The incomplete understanding of fairness metrics hinders the development of specific definitions and recommendations in reporting guidelines, such as TRIPOD+AI,12 the Fairness, Universality, Traceability, Usability, Robustness, and Explainability-AI (FUTURE-AI) consensus guideline for trustworthy and deployable AI in health care,46 and the Standards for Data Diversity, Inclusivity, and Generalisability (STANDING Together) consensus recommendations.30 These guidelines address fairness considerations with caution and minimal specificity. Although previous notable efforts have reviewed fairness in machine learning,36,47–51 they rarely focus on CPMs or do not provide a comprehensive or critical perspective on fairness metrics in CPMs.45,52,53 Hence, we sought to identify and critically appraise fairness metrics reported in the CPM literature by addressing the following questions: first, which fairness metrics have been proposed, applied, and analysed in the CPM literature; second, how each metric should be interpreted in light of existing ethical and legal frameworks; and third, when the use of each metric is justified. Methods We conducted a scoping review to identify fairness metrics used in the clinical predictive AI literature. The inclusion criteria and article selection process were kept intentionally broad to maximise coverage. We followed the methodological framework by Arksey and O’Malley54 and adhered to the PRISMA extension for Scoping Reviews (PRISMA-ScR) guidelines.55 For the purpose of this scoping review, we define fairness metric as a metric that quantifies the extent to which a model’s output does or does not discriminate (in the societal sense, based on a given notion of fairness) against individuals or groups defined by a sensitive attribute (table 1).56 Search strategy and selection criteria JM searched the literature published between Jan 1, 2014, and Oct 22, 2024, in PubMed, ACM Digital Library, IEEE Xplore, arXiv, and medRxiv databases to include journal publications, conference publication, and preprints. The search was limited to studies published in English. Four main keywords—namely, “fairness”, “metric”, “clinical”, and “model”—and their equivalent terms (“fair*”; “metric*”; “health*”, “clinic*”, “medic*”; “model*”, “algorithm*”, “machine learning”) were searched for in the title or abstract of articles. We deliberately excluded the terms “bias”, “parity”, and “disparity” from our search strategy owing to their ambiguity and broad use across different disciplines. Complete details of search strings for each database are previously published.45 Backward citation tracking was conducted throughout the review period from Oct 22, 2024, to May 1, 2025. Non-duplicate research studies and review articles that directly engaged with fairness considerations in applied CPMs were included without restrictions on how fairness was addressed. We excluded studies if the concerned metrics did not align with our definition of a fairness metric (table 1). We also excluded causal fairness notions (not necessarily the complete document) that require counterfactual predictions. Metrics that were devoid of sufficient detail to permit clear reporting of their definitions were also excluded. Finally, studies that did not propose or define fairness metrics were removed. To reduce the risk of omission, we complemented the database search with backward citation tracking and manual reference screening to finally identify 63 distinct fairness metrics. These steps helped to include metrics from previous non-clinical reviews and those from new studies, relevant to CPMs. Backward citation searching was conducted after screening based on inclusion and exclusion criteria to maximise the identification of relevant fairness metrics and to obtain additional details on their definitions. This approach enabled the inclusion of both review articles and original studies that proposed, defined, or applied fairness metrics, although these metrics were not originally developed for health-care contexts. Data extraction: metric information For each metric, JM extracted the formula and all information required for its computation. In case of studies proposing families of metrics, we report specific representative examples as applicable (eg, equity-scaled area under the receiver operating characteristic curve [AUROC] for the family of equity-scaled metrics57). Metrics with ambiguous names were standardised to ensure consistency and alignment with the information in the literature. For example, the metric discrimination index was redefined as F1-score parity after reviewing the implementation details of both metrics.58 We categorised all metrics based on their proposed domain (health care or others) and their positions within existing fairness taxonomies.1,31,36,39 We also identified alternative names of metrics and the required inputs for their computation (eg, type of model output, outcome labels, or predictor conditioning). Further, we examined the types of sensitive attributes (binary, multigroup, or continuous) that each metric supports. We also assessed interpretation aspects, including bias-preserving versus bias-transforming properties,1,31,36,39 outcome prevalence dependency, and target fairness values. Data extraction: taxonomy of fairness metrics Multiple and conflicting fairness taxonomies (n=12)45 exist in the literature. The figure presents the taxonomy adopted in this scoping review for fairness metrics. The adopted taxonomy builds on previous taxonomies and can serve as a guide in selecting metrics according to the available information and fairness objectives. Evaluation of fairness begins with consideration of performance dependency—that is, whether fairness should be evaluated with respect to model performance (performance-dependent or supervised fairness evaluation) or not (performance-independent or unsupervised fairness evaluation, with no consideration of outcome labels or ground truth labels). This distinction between the evaluation approaches hinges on the interpretation of fairness and validity of the labels used to develop and evaluate the model. If these outcome labels are accepted as a fair representation of the target population, the use of a performance-dependent metric is justified. Conversely, if the labels risk encoding health inequities that the model should not reinforce (eg, disparities in access to care), performance-independent metrics are more appropriate. In practice, what is being inquired is whether the status quo reflected in the data and historical outcome labels are acceptable as a reference (figure). The second consideration is whether fairness should be assessed at the level of estimated probability P^ or the predicted class Y^. Probability-based metrics (eg, mean score parity59) assess fairness without applying a decision threshold. In contrast, threshold-dependent metrics (eg, statistical parity60) evaluate fairness at the predicted class level; the application of threshold-dependent metrics thus depends on applying a threshold to convert the estimated probability into class prediction. This distinction has also been described in a preprint published online in 2025 as approaches that use “informing with a risk score” (probability-based evaluation) versus “decision support with a classification” (threshold-dependent evaluation).61 The third consideration is the type of performance metric. If a performance-dependent fairness metric is chosen, the focus shifts to what type of model performance1 should be compared across groups. Probability-based metrics focus on calibration (eg, calibration-in-the-large parity62), discrimination (eg, AUROC parity63), or overall performance (eg, Brier score parity64), whereas threshold-dependent metrics focus on partial metrics (eg, equality of opportunity difference42), summary metrics (eg, accuracy gap65), or clinical utility (eg, subgroup net benefit66). Following prior taxonomies, fairness metrics were also categorised as individual39 or group fairness metrics36 (table 1). Data extraction: qualitative critical appraisal We developed a data extraction form45 to critically appraise each metric by assessing its suitability, limitations, and potential pitfalls in clinical scenarios. The appraisal process included outlining its justified use. We provide overall critical appraisal and guidance for different groups of fairness metrics. To aid interpretation, we categorised our recommendations into three levels of guidance: recommended if, referring to metrics that are generally suitable and align with clinical and ethical standards under specific conditions; use with caution, referring to metrics that might be relevant but require careful contextual consideration and are not essential; and inadvisable, referring to metrics with substantial limitations or risk of harm that outweigh potential benefits in most scenarios. Results 927 documents were identified through database searches. After removing duplicates and ineligible records based on language and format, 820 records were screened for inclusion. 644 articles were excluded for not focusing on CPMs, not addressing fairness, or being devoid of any fairness metric. The remaining 176 reports were sought for retrieval, with 157 further excluded during data extraction owing to the absence of fairness metric proposals, definitions, or relevant applications. Finally, 19 reports met inclusion criteria, and an additional 23 reports were identified through backward citation searches, resulting in a total of 42 articles (three reviews and 39 research articles) describing fairness metrics relevant to CPMs.45 Across the 42 papers, we identified 63 fairness metrics that aligned with the definition proposed in this Review (table 1). A comprehensive and detailed formalisation (including mathematical formulation) of each metric have been previously published.45 Fairness metrics from most articles (20 of 42) originated from AI rather than from biomedical (14 of 42) or applied ethics (eight of 42) research. Only 19 of the 63 metrics were explicitly defined for health-care applications. Metrics identified Among the identified performance-independent metrics (15 of 63, including that reported in a preprint67),48,60,67–73 most (n=13) were group fairness metrics, and only two were individual fairness metrics. The performance-independent metrics were further divided into probability-based (n=6, including mean score parity59) and threshold-dependent (n=7, including statistical parity60). However, only three of these 15 metrics were proposed for health-care applications. Performance-dependent metrics (n=48, including those reported in preprints66,74)42,51,57,58,62–66,69,74–96 were substantially more common than performance-independent metrics, and all studies associated with performance-dependent metrics focused on group fairness. Only 16 of these 48 metrics were explicitly proposed for use in health-care contexts. The performance-dependent metrics were also divided into probability-based (n=22) and threshold-dependent (n=26) metrics. Most of the probability-based performance-dependent metrics focused on discrimination (n=12, including AUROC parity63); these metrics were followed by calibration-based metrics (n=5, including expected calibration error parity78) and metrics that focused on overall performance (n=5, including Brier score parity64). Threshold-dependent performance-dependent metrics primarily assessed fairness using confusion matrix-derived metrics, with most of them focusing on partial metrics (n=18, including equal opportunity difference42), followed by those focusing on summary metrics (n=7, including overall accuracy gap65). Only one threshold-dependent performance-dependent metric that focused on clinical utility (subgroup net benefit66) was identified (table 2). Most of the metrics identified (61 of 63) were group fairness metrics. Individual fairness metrics were notably scarce (two of 63). A structured catalogue of fairness metrics, including classification information within the fairness taxonomies, operationalisation details, interpretation and implications, and justified use and critical appraisal, is summarised.45 Critical appraisal of identified fairness metrics Based on the catalogue of fairness metrics, we critically appraised the metrics in terms of their applicability, interpretability, quality of definition, validation, and alignment with clinical and ethical considerations (table 3). Table 4 lists examples of metrics that are currently used in real-world scenarios. We summarise key qualitative findings for each category, providing guidance on which metrics to prioritise when assessing the fairness of CPMs. Performance-independent metrics typically aim to ensure parity in the model’s positivity rates.31 For these metrics, a perfectly accurate model (ie, with no classification errors at the population level after finalising a decision threshold) might never achieve complete fairness if prevalence differs across groups defined based on the sensitive attribute.102 Probability-based performance-independent metrics (eg, mean score parity59) should be used with caution as they are challenged by their reliance on mean values without a defined metric of uncertainty. Metrics such as unsupervised ranking fairness69 might be of value if some form of unsupervised discrimination analysis was deemed necessary; however, these metrics are inadvisable without further empirical evaluation, as their theoretical foundation in health care remains unclear with no proper validation. Threshold-dependent performance-independent metrics such as statistical parity,60 children metrics,68,72 and disparate impact73 can be useful for assessing fairness at the predicted class level, without relying on the quality of existing labels. Conditional statistical parity48 metric is especially relevant when legitimate factors (such as comorbidities or biological differences) justify group-specific, differential risk estimates. The disparate impact metric has been connected with the US disparate impact law73 despite the fact that the four-fifths rule that is often associated with this metric is neither necessary nor sufficient to abstract the disparate impact law;112 in contrast, the conditional statistical parity metric aligns with the EU non-discrimination law, which could be desirable from a legal perspective.97 However, operationalising fairness remains inconsistent when sensitive attributes are non-binary or require conditioning across multiple categorical strata. Although survival70 and censoring-based71 metrics extend fairness evaluation to more complex scenarios than standard prediction settings without time-to-event outcomes, the adoption of these metrics is hindered by insufficient empirical evaluation and understanding. Similarly, metrics such as differential fairness68 are not supported by adequate empirical evidence and are currently inadvisable for real-world use. Performance-dependent metrics compare performance across individuals or subgroups of individuals. According to these metrics, a model is perfectly fair when it achieves, for example, consistent (but not necessarily perfect) discrimination, calibration, or clinical utility across subgroups. These metrics assume that the distribution of the predictors and outcomes present in the model development data reflect the nature of target population. By prioritising consistency in model performance (ie, parity across individuals or groups), the status quo in health-care delivery is preserved and labels are reinforced.31 Whether statistical parity should be prioritised in fairness evaluation remains debated; some researchers argue that model performance can naturally vary between subgroups and that the focus should instead be on ensuring minimum acceptable performance for all subgroups.102 Probability-based performance-dependent metrics assess disparities in model performance across subgroups using base performance metrics that are probability-based and do not apply thresholds. These metrics are often preferred for CPMs because they evaluate model performance independently of threshold choices that might differ across subgroups or clinical settings.1 Among probability-based performance-dependent metrics focusing on discrimination, AUROC parity,63 which was built on a previous study1 is recommended for quantifying discrimination disparities. Although AUROC, with its value remaining unchanged after rank-preserving transformation of estimated probabilities, is a semi-proper metric, it remains useful as it focuses on discrimination performance. In contrast, area under the precision–recall curve parity64 metric is inadvisable owing to its semi-proper scoring nature, absence of focus (mixing discrimination and clinical utility),1 and tendency to favour groups with higher outcome prevalence.98 Extensions such as xAUROC,76 xAUROC disparity,76 sAUROC,77 pairwise ranking fairness,69 mean performance-scaled disparity AUROC (reported in a preprint74), equity-scaled AUROC,57 concordance imparity,79 and AUROC parity ratio80 are not backed by any clear motivation, sufficient implementation details, or in-depth clinical validation, making them challenging to interpret and inadvisable for use in the time being. Among probability-based performance-dependent calibration metrics, equal calibration82 and well calibration51 are vaguely defined in the literature and should be used with caution, particularly as no clear implementation details exist on how to consider multiple thresholds. Nevertheless, such metrics that satisfy the notion of equal calibration across groups are (in principle) desirable.99 Expected calibration error parity,78 calibration-in-the-large parity,62 absolute calibration error parity,81 and other similar metrics should also be used with caution. Although such metrics can be informative to assess calibration, their use should be accompanied by a calibration plot for each subgroup and be paired with discrimination metrics. Furthermore, despite being proper metrics,1 expected calibration error parity and absolute calibration error parity metrics have been criticised for being dependent on the way in which calibration binning is performed and for handling overestimation and underestimation in a similar manner.100 Behaviour of fairness metrics focusing on differences in expected calibration error parity or absolute calibration error parity is not well studied. Among probability-based metrics that focus on overall performance, Brier score parity64 and log-loss parity81 metrics should be used with caution. Despite being proper metrics (table 1) of model performance,1 these two metrics offer restricted interpretability and provide only an overall assessment of the model, blending elements of discrimination and calibration.1 Balance for positive class and balance for negative class metrics82 should also be used with caution because they are not studied for use in CPMs. The Earth mover’s distance for equivalent separation,64 which compares the distributions of predicted outcome probabilities using the Wasserstein distance, can be conceptually informative as it seeks to satisfy the separation non-discrimination criterion of fairness36 in its strongest form. Although its behaviour in CPMs is expected to be appropriate, the metric is not empirically evaluated. Threshold-dependent performance-dependent metrics compare classification performance across groups. These metrics are inherently constrained because they rely on a decision threshold that might be chosen with no clinical rationale. Consequently, the reported quantity (eg, accuracy) is specific to the chosen threshold and might not be generalisable for other metrics. Moreover, some of these metrics are improper when considered alone.1,98 Among threshold-dependent performance-dependent metrics focusing on partial metrics, equal opportunity difference42 and predictive parity,87 which aim to achieve parity in true-positive and false-positive rates, can be relevant if reported together. Similar considerations apply to predictive parity87 and negative predictive value parity.75 These metrics have the advantage of being easily interpretable; however, their reliance on potentially arbitrary thresholds restricts standalone use. Equalised odds42 is inadvisable because combining it with true-positive and false-positive rates makes the metric lose focus, hampering a clear interpretation. Worst-group recall91 and maximum difference recall89 should be used with caution. They might be relevant under Rawlsian fairness principles, which prioritise improving outcomes in the worst-off subgroup, but their implications require careful evaluation. Recall intergroup standard deviation,92 recall coefficient of variation,64 recall disparity ratio,90 and recall between-group generalised entropy index93 are highly complex as metrics, insufficiently studied, and difficult to interpret. Finally, Recall-HEAL94 introduces reparative fairness by computing an anticorrelation between model performance and historical disease burden. However, complex tradeoffs and ethical debates might arise, and the use of external data could be deemed arbitrary, making the use of this metric and related metrics questionable. Among threshold-dependent performance-dependent metrics focusing on summary metrics, overall accuracy gap,65 error rate ratio,95 and balanced accuracy difference75) are inadvisable owing to improper metric behaviour (table 1), paucity of error-type distinction, and threshold dependency. F1 parity58 is inadvisable owing to improper scoring behaviour. The use of Matthew’s correlation coefficient parity96 is also questionable because of its highly complex formulation that hampers interpretability. The treatment equality65 metric has unclear rationale and implications. Among metrics focusing on clinical utility metrics, only one metric—subgroup net benefit—reported in a preprint,66 was identified. Although further empirical evaluation is needed, this metric can be particularly relevant,1,103,105 as it reflects the quality of decision making (at a clinically meaningful threshold) and captures the tradeoff between potential harms and benefits. Net benefit is defined as sensitivity × prevalence – (1 – specificity) × (1 – prevalence) × w; here, w is the odds at the threshold probability. Net benefit is lower for subgroups with lower prevalence, reflecting expected differences.103,105 Subgroup net benefit, as defined by Benitez-Aurioles and colleagues,66 allows each subgroup to have its own outcome prevalence term, accounting for pre-existing prevalence differences or health inequities. Discussion In this scoping review, we identified and examined 63 fairness metrics used in CPMs. We evaluated how each metric should be interpreted in light of relevant ethical and legal frameworks and assessed when the use of each of these metrics is justified. We carried out a qualitative critical appraisal and provided practical guidance that considers the implications, limitations, and appropriate contexts for each metric. The panel provides broader recommendations for the fairness evaluation of CPMs, emphasising that fairness evaluation is not restricted to the choice of fairness metrics and that transparent reporting is fundamental for fairness evaluation of CPMs. Our scoping review revealed broader issues regarding fairness evaluation that warrant discussion. The definition of fairness metric in the literature is often vague or ill defined. This conceptual ambiguity undermined our ability to systematically assess the fairness of CPMs. To address this gap, we proposed a working definition of fairness metric, which also serves as the foundation for our methodology. Although diversity in methodological approaches can be beneficial, the high number of proposed metrics and definitions can result in redundancy and overlap, with subtle variations in their formulations leading to inconsistent interpretation of results. Fairness metrics largely originate from computer science39 and are frequently evaluated in case studies unrelated to health care, such as recidivism prediction99 or credit scoring.42 These metrics often emerge in isolation or is at risk of epistemic trespassing112 in the absence of a clear ethical or theoretical foundation,102 raising concerns about the suitability of these metrics for clinical applications.113 Operational challenges increase as fairness metrics become more complex. Once fairness assessments move beyond binary classification tasks, binary sensitive attributes, or simple conditioning approaches, computing and interpreting these metrics becomes non-trivial. We found that fairness metrics are often devoid of clear definitions, justified use, or sufficient empirical evaluation, restricting their reliability for real-world AI-based CPMs. Many metrics suffer from theoretical ambiguities, potentially inconsistent behaviour, or inadequate empirical support. The level of scrutiny applied for different metrics varies widely. Some metrics are rigorously studied and grounded in ethical or legal principles (eg, equality of opportunity difference42), whereas others are introduced with minimal justification (eg, equity-scaled AUROC57). The predominance of performance-dependent (48 of 63) and threshold-dependent (33 of 63) metrics further underscores a methodological convenience bias. These metrics are easier to compute than performance-independent and probability-based metrics but might not capture a full and relevant picture of model behaviour. Performance-dependent metrics might preserve existing biases in the data rather than questioning them,31 whereas threshold-dependent metrics might be inherently inadequate as they often assess fairness at arbitrarily chosen decision cutoffs.32 Individual fairness metrics were notably scarce (two of 63), largely because individual metrics are difficult and sometimes controversial to operationalise. We recommend readers to refer to the study by Anderson and Visweswaran52 for more on individual fairness metrics. However, the authors52 found only six uses of individual fairness metrics in health care,71–118 and their operationalisation as metrics remains unclear. Beyond group and individual fairness, intersectionality presents an important consideration in fairness assessment. Intersectionality captures how overlapping social identities interact to create unique experiences of privilege or oppression.119 Although metrics can be operationalised as group fairness metrics with varying subgroup granularity, intersectional or subgroup families of fairness metrics have been proposed.68,120 Existing fairness metrics often assess disparities along single attributes such as sex, age, or race, whereas real-world inequities arise at the intersections of these dimensions. For example, an 80-year-old Black woman might face disadvantages not captured by separate assessments of age, race, or sex.121 Another notable gap in the literature was the restricted presence of clinical utility metrics. Only one metric, subgroup net benefit,66 was explicitly defined as capturing clinical utility, despite the importance of such metrics for informing decision making.1 Two other metrics—equalising disincentives84 and treatment equality65—appeared to draw on utility reasoning; however, both of them were neither explicitly framed as clinical utility metrics nor originally proposed in a health-care context. Existing fairness metrics focus largely on parity in statistical attributes but do not expose any downstream effect on patient outcomes, which is arguably the most important aspect for models in a clinical setting. Implementing predictive AI models should ultimately translate into improved patient outcomes; therefore, future research should prioritise metrics that explicitly link fairness to clinical utility.122 For instance, subgroup-specific decision curves and net benefit-based fairness metrics could be of substantial value.123 Most fairness metrics are parity based and primarily capture numeric discrepancies between groups. However, these discrepancies do not necessarily correspond to clinically meaningful disparities. The interpretation of such discrepancies depends heavily on the performance regimen of the model. For instance, a difference of 0·10 between AUROC values of 0·90 and 0·80 across subgroups might have very different clinical implications than implications from the same difference between values of 0·70 and 0·60. This variability raises important considerations regarding when a disparity should prompt concern and whether statistical significance alone can be a basis for intervention. Rather than aiming for statistical parity at all costs, fairness should be framed in terms of minimum acceptable performance for all groups,102 guided by clinical relevance and known health disparities. The downstream effects of fairness violations on CPM deployment is inherently context-dependent. For example, in scenarios in which all groups benefit from the model, the fact that one group benefits less than other groups might still be problematic. Although such a scenario could represent an improvement from previous standard practice, tolerating such disparities might have long-term implications. Once deployed, models are subject to data drift and performance degradation. Thus, groups that initially benefited less might be the first to experience a decline in performance, a phenomenon that has been described as fairness drift.109 The following crucial gaps were identified in fairness metric development and evaluation during our scoping review. Sample size requirements remain largely unaddressed during fairness evaluation.124,125 Fairness metrics rarely report confidence intervals to quantify statistical uncertainty; however, these intervals are fundamental for interpretation as groups typically have very different (and small) sample sizes.126 Next, intersectionality remains insufficiently addressed, as most fairness assessments treat sensitive attributes independently and overlook interactions across multiple axes of disparity.40 Intersectional analyses are methodologically challenging because of small subgroup sizes, even in large datasets.124,127–129 Clinical utility is rarely studied, despite its importance.102 Furthermore, the statistical behaviour of many fairness metrics are not empirically evaluated in medical settings, hampering their reliability and usefulness. Finally, tradeoffs between fairness metrics remain understudied for CPMs although some metrics are mutually incompatible.82 Future works should prioritise fairness metrics that provide uncertainty estimates (eg, by using bootstrap or closed forms when available), support intersectional analyses, align with clinical outcomes and benefit, and are systematically evaluated across real-world health-care settings. We did not extensively assess the practical applications of fairness metrics in the literature. Future research should translate our qualitative critical appraisal into quantitative evidence via simulation studies or using real-world data and thereby support recommendations for selecting and applying fairness metrics in CPMs. Conclusion The current landscape of fairness metrics in clinical predictive AI is fragmented, characterised by unclear definitions, standardisation, and clinical relevance. Several metrics are insufficiently evaluated empirically, hampering their real-world relevance. Future research should prioritise fairness evaluations that align with clinical decision making, incorporate uncertainty estimation, and account for intersectionality. Fairness evaluations should be conducted contextually and collaboratively with key actors. Supplementary Material