Signal Fidelity Index-aware calibration for addressing distributional shift in predictive modeling across heterogeneous real-world data Authors: Jingya Cheng, Jiazi Tian, Federica Spoto, Alaleh Azhir, Daniel Mork, Hossein Estiri Published: 2025 ABSTRACT Machine learning models trained on real-world data (RWD) often experience performance degradation when deployed across different settings due to distributional shift. However, a fundamental but under-explored factor contributing to this degradation is the decay of diagnostic signals: systematic variability in diagnostic quality and consistency across institutional contexts, which affects the reliability of clinical codes used for model training and prediction. To develop and evaluate a Signal Fidelity Index (SFI) that quantifies diagnostic signal decay at the patient level across diverse clinical conditions, and to assess the effectiveness of SFI-aware calibration in improving model performance compared to established calibration methods, without requiring outcome labels in target domains after initial method development. We developed a comprehensive simulation framework using synthetic patient datasets across six clinically diverse phenotypes: dementia, geriatric bipolar disorder, fibromyalgia, adult ADHD, type 2 diabetes, and hypertension. Each phenotype included independent simulation batches with varying demographic compositions and data quality characteristics. The SFI was constructed from six components: diagnostic specificity, temporal consistency, entropy, contextual concordance, medication alignment, and trajectory stability. We implemented SFI-aware calibration using a multiplicative adjustment formula with phenotype-specific calibration parameters optimized through supervised parameter development, then evaluated performance in label-free deployment across heterogeneous testing datasets. We compared SFI-aware calibration against established baseline calibration methods. SFI-aware calibration significantly improved predictive performance against both uncalibrated predictions and all baseline methods across nearly all six phenotypes (Cohen’s d = 0.603–5.002, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p < 0.001$$\end{document}). Performance improvements varied by phenotype complexity, with F1-score gains ranging from +4.2% for fibromyalgia to +34.9% for dementia, and AUC gains ranging from +4.7% to +40.1%. Traditional calibration methods often led to degraded performance, with isotonic regression exhibiting universal failure (AUC values degraded to 0.51–0.56 across all phenotypes) and Platt scaling demonstrating inconsistent, phenotype-dependent effects. Brier score decomposition revealed that SFI-aware calibration improved performance through a dual mechanism: reliability reductions ranging from 11% to 29% and resolution increases ranging from +35% to +238%. Notably, even well-diagnosed conditions with standardized diagnostic criteria (type 2 diabetes, hypertension) showed substantial benefits (+8.0% to +13.3% F1-score, +25.1% to +40.1% AUC), suggesting that diagnostic signal variability affects all EHR-based phenotyping. These findings demonstrate that diagnostic signal decay is a tractable problem that can be systematically addressed through patient-level fidelity-aware calibration strategies. SFI-aware calibration offers a practical approach for enhancing the performance of clinical prediction models across diverse healthcare contexts, requiring supervised parameter development once per phenotype, followed by label-free deployment to unlimited target populations. The method consistently outperforms established calibration techniques while avoiding their tendency to degrade performance, making it particularly suitable for large-scale administrative datasets where outcome labels are unavailable. Multi-phenotype validation confirms generalizability across clinical conditions ranging from complex, under-diagnosed phenotypes to well-diagnosed conditions with standardized criteria. Introduction Predictive models trained on structured real-world data (RWD), such as electronic health records (EHRs) or insurance claims, often achieve strong performance within the environments in which they are developed. Yet when applied across institutions or patient populations, their reliability frequently deteriorates. This breakdown reflects more than a conventional generalization problem. It arises from the systematic variability of the diagnostic signals themselves, which degrade as they are recorded, encoded, and transmitted across institutional contexts, a phenomenon characterized by Spoto et al. (2015) as diagnostic signal decay1. Diagnostic code heterogeneity across healthcare settings is well-documented in the literature2,3. An illustrative case is dementia diagnosis codes. In some healthcare systems, such codes represent the culmination of detailed clinical evaluations, including longitudinal testing, neuroimaging, and specialist review. In others, the same codes may reflect provisional judgments or administrative expediency driven by billing requirements4,5. As a result, diagnostic codes capture not only disease biology but also institutional practices, economic pressures, and the heuristics of human decision-making. What appears to be a uniform data element is, in reality, a heterogeneous, context-dependent signal. This variability presents a central challenge for machine learning in healthcare. Phenotyping and risk prediction models are susceptible to such instability, and performance declines are well-documented when models are transferred between domains6–8. Domain adaptation and distributional shift are extensively studied phenomena in machine learning9,10, with existing strategies including domain adversarial training, instance reweighting, and recalibration that typically treat diagnostic variability as noise to be removed. These methods also rely on labeled target-domain data, continuous retraining, and significant computational resources, making them difficult to implement in practice11,12. Our contribution lies not in discovering that diagnostic codes vary across settings, but rather in systematically quantifying this variability at the patient level and integrating it into a practical calibration framework. We introduce the Signal Fidelity Index (SFI), a composite metric that quantifies the reliability of diagnostic data across three dimensions: the internal consistency of diagnostic assertions (clarity), the alignment of codes with expected clinical trajectories (temporal coherence), and the correspondence of diagnoses with established patterns of disease presentation (contextual alignment). Computed entirely from structured EHR data, the SFI operationalizes tacit clinical knowledge about diagnostic reliability into six computable components measurable from routine healthcare data. Building on this, we present SFI-aware calibration, a lightweight, post hoc method that adjusts model prediction confidence according to the fidelity of the underlying data. Unlike many adaptation methods, SFI-aware calibration employs a hybrid framework: supervised parameter development determines optimal calibration strength once per phenotype using labeled validation data, followed by label-free deployment to unlimited target populations without requiring outcome labels. This distinguishes our approach from traditional calibration methods such as Platt scaling and isotonic regression, which require labeled data at every new deployment setting, limiting scalability to diverse healthcare contexts. To evaluate generalizability and address potential concerns about phenotype specificity, we conducted a comprehensive evaluation across six clinically diverse conditions, spanning complex, underdiagnosed phenotypes with high diagnostic variability (dementia, geriatric bipolar disorder, fibromyalgia, and adult ADHD) to well-diagnosed conditions with standardized diagnostic criteria (type 2 diabetes and hypertension). Furthermore, we benchmark SFI-aware calibration against five established calibration methods (Platt scaling, isotonic regression, beta calibration, temperature scaling, histogram binning) to rigorously assess the value added by fidelity-aware adjustment. This comparison reveals that SFI-aware calibration consistently improves performance without ever degrading below an uncalibrated baseline, suggesting fundamental advantages of patient-level fidelity weighting over uniform calibration approaches. By operationalizing diagnostic signal decay1 and introducing a method to quantify and account for it, this work reframes data reliability as a first-class consideration in predictive modeling. Our experiments demonstrate that SFI-aware calibration simultaneously enhances calibration quality and improves discrimination ability. The approach offers a practical pathway toward more transferable machine learning systems in clinical settings, with clear implications for large-scale deployment across administrative datasets where outcome labels are prohibitively expensive or impossible to obtain. Background In label-free calibration contexts, where ground-truth outcomes are unavailable in the target domain, several techniques have been developed to address distribution shifts and enhance model reliability. For instance, LaSCal provides a consistent calibration error estimator under label shift by reweighting the source label distribution using unlabeled target data, enabling post-hoc calibration without target labels14. Similarly, in-context comparative inference for large language models (LLMs) improves zero-shot and few-shot calibration by incorporating unlabeled samples into prompts, aggregating probabilities to mitigate indiscriminate miscalibration and enhance F1 scores and accuracy. Confusion matrix estimation methods, such as those using Average Threshold Confidence (ATC) and Difference of Confidences (DoC), predict performance metrics like Precision, Recall, and AUC from unlabeled target confidences, offering robust out-of-distribution evaluation. Reweighting approaches for conformal prediction under label shift adjust calibration sets using estimated shift factors from unlabeled data, ensuring theoretical coverage guarantees in distribution-free settings. Evidential deep learning further refines pseudo-labels in source-free domain adaptation by modeling predictive uncertainty without source data or labels, reducing overconfidence through Dirichlet priors15. These methods collectively enable practical uncertainty quantification across domains like healthcare and imaging. However, existing label-free calibration techniques, such as LaSCal or conformal reweighting, assume specific shifts (e.g., label or covariate) but overlook institutional variability in diagnostic quality16,17. Additionally, in large-scale unlabeled datasets, methods such as in-context comparative inference or confusion matrix estimation may falter due to noisy or inconsistent signals18. Methods Overview and study design We developed and evaluated a Signal Fidelity Index (SFI)-aware calibration method designed to improve prediction performance across heterogeneous datasets without requiring outcome labels in target populations. The method operationalizes the concept of “diagnostic signal decay”1, i.e., systematic deterioration in data quality across different healthcare settings with varying population characteristics and healthcare utilization practices – by quantifying patient, level diagnostic fidelity using structured EHR metadata alone. Our evaluation consisted of a large-scale simulation study across six distinct clinical phenotypes (dementia, geriatric bipolar disorder, fibromyalgia, adult ADHD, type 2 diabetes, and hypertension), generating 3,550 synthetic electronic health record (EHR) datasets and enabling systematic assessment under controlled conditions that mimic real-world diagnostic variability. We distinguish between two phases of SFI-aware calibration to clarify when outcome labels are required and when they are not. The first phase, supervised parameter development, requires labels and involves determining the optimal calibration strength parameter (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}) once per phenotype using labeled validation data from a single reference cohort. This phase is analogous to how traditional calibration methods, such as Platt scaling and isotonic regression, determine their parameters. For each phenotype, we generate reference datasets with known outcome labels, train prediction models on reference training data, systematically test multiple \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values ranging from 0.5 to 2.5, select optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} by maximizing the geometric mean of F1-score and Recall improvements (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{\Delta F1 \times \Delta \text {Recall}}$$\end{document}), and validate statistical significance (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p < 0.05$$\end{document}) of performance gains. The second phase, label-free deployment, requires no outcome labels in target populations. Once \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} is established for a phenotype, such as dementia, SFI-aware calibration can be deployed to unlimited unlabeled target populations using only the SFI calculation derived from structured EHR metadata (diagnosis codes, medications, care settings, encounter dates), the reference SFI mean (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{SFI}_{\text {ref}}$$\end{document}) from the development cohort, and the established \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} parameter for that phenotype. This hybrid framework distinguishes SFI-aware calibration from purely supervised methods, which need labels at both development and deployment, while acknowledging the necessity of labeled data during initial method development. In real-world deployment, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} would be determined once using a labeled cohort, such as a research cohort with comprehensive psychiatric evaluations, and then applied across multiple unlabeled datasets, including different hospital systems and Medicare claims, without requiring labeled outcomes in each new setting. Phenotype selection and rationale To evaluate generalizability and address the concern that results might be specific to dementia alone, we conducted a multi-phenotype evaluation spanning six clinically distinct conditions. Phenotypes were strategically selected to represent a spectrum from complex, underdiagnosed conditions with high diagnostic variability (where SFI-aware calibration is expected to provide the greatest benefit) to well-diagnosed conditions with standardized diagnostic criteria (where benefits are expected to be modest). Table 1 summarizes phenotype characteristics.Table 1Clinical characteristics of evaluated phenotypes.PhenotypeGroupDiagnostic delayCode heterogeneityExpected SFI benefitDementiaComplex2–3 yearsHighHighGeriatric bipolarComplex5–10 yearsHighHighFibromyalgiaComplex5–7 yearsHighHighAdult ADHDComplex10–15 yearsHighHighType 2 diabetesWell-diagnosedMinimalLowLowHypertensionWell-diagnosedMinimalVery LowMinimal Table 1Clinical characteristics of evaluated phenotypes.PhenotypeGroupDiagnostic delayCode heterogeneityExpected SFI benefitDementiaComplex2–3 yearsHighHighGeriatric bipolarComplex5–10 yearsHighHighFibromyalgiaComplex5–7 yearsHighHighAdult ADHDComplex10–15 yearsHighHighType 2 diabetesWell-diagnosedMinimalLowLowHypertensionWell-diagnosedMinimalVery LowMinimal Clinical characteristics of evaluated phenotypes. Dementia was selected because approximately 50% of cases remain undiagnosed13, the condition requires specialist evaluation, exhibits substantial coding heterogeneity ranging from specific diagnoses such as G30.x Alzheimer’s disease to nonspecific codes like F03 unspecified dementia, and has highly specific medications (e.g., memantine). Geriatric bipolar disorder is severely underdiagnosed in older adults, requires psychiatric specialist evaluation, exhibits high code variability due to comorbid conditions, and uses specific mood stabilizers and antipsychotics. Fibromyalgia presents with a long diagnostic odyssey averaging 5–7 years delay, functions as a diagnosis of exclusion requiring rheumatology referral, shows highly heterogeneous coding patterns ranging from M79.7 fibromyalgia to nonspecific myalgia codes, and has highly specific medications such as pregabalin and duloxetine for fibromyalgia indication. Adult ADHD is extremely underdiagnosed in adults, requires a comprehensive neuropsychiatric evaluation, presents variable diagnostic codes, including F90.x ADHD versus attention difficulties, and involves controlled substance medications such as stimulants that require careful documentation. Compared to dementia, Type 2 diabetes represents a well-diagnosed condition owing to objective laboratory thresholds (e.g., HbA1c \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge$$\end{document} 6.5%), standardized diagnostic codes using E11.x, and widely used nonspecific medications such as metformin. Hypertension has variable diagnosis rates, requires multiple elevated readings \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge$$\end{document} 130/80 or \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ge$$\end{document} 140/90 mmHg, uses multiple ICD-10 codes (I10-I15) for different types, and is treated with specific drug classes (ACE inhibitors, ARBs, calcium channel blockers, diuretics, beta-blockers) tailored to individual patients. This phenotype selection strategy enables the assessment of whether SFI-aware calibration provides a substantial benefit for complex phenotypes with high diagnostic signal decay, while maintaining stability and avoiding performance degradation where signal decay is minimal. Simulation framework For each phenotype, we simulated 10 independent batches, with each batch containing one reference dataset of 2,000 patients for model training and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} optimization along with 50 testing datasets of 1,000 patients each exhibiting varying demographic compositions and data quality. This design yielded 3,000 total testing datasets across six phenotypes. Each phenotype utilized clinically realistic parameters adapted from epidemiological literature. Disease prevalence was set to published estimates for reference datasets, while testing datasets varied by ±10 percentage points. Age distributions reflected phenotype-appropriate ranges, including ages 65–85 years for dementia, 25–55 years for adult ADHD, and 40–80 years for hypertension. Risk models incorporated age and race effects calibrated to published relative risks for each condition. Diagnostic codes comprised phenotype-specific high-fidelity and low-fidelity ICD-10 codes curated by clinical experts. Medications reflected condition-specific prescriptions with realistic patterns, and care settings captured inpatient versus outpatient utilization rates matched to condition acuity. Complete phenotype parameter specifications are provided in Supplementary Materials. Demographic generation, outcome assignment, diagnosis code allocation, medication prescription, and care setting assignment followed the framework described for dementia in the original manuscript, with adaptations using phenotype-specific parameters. Key shared features included assignment of 2–20 healthcare encounters over five years for each patient, outcome labels approximated from epidemiologically-validated framewords, high-fidelity codes weighted twice as heavily for diagnosed patients, low-fidelity codes assigned at 10–30% base rates for non-diagnosed patients, phenotype-specific medications aligned with diagnosis status, and care setting patterns reflecting disease acuity. Signal Fidelity Index (SFI) The SFI is computed as the arithmetic mean of six normalized components (on a 0–1 scale), quantifying diagnostic reliability using only structured metadata. Table 2 provides operational definitions; Appendix A contains complete mathematical formulations.Table 2Signal Fidelity Index (SFI) components.ComponentDefinitionSpecificityProportion of diagnosis codes that are specific and definitive (e.g., G30.9 Alzheimer’s vs F03 unspecified dementia)Temporal ConsistencyOne minus the frequency of diagnosis code switching across sequential encountersEntropyInverse of normalized Shannon entropy of diagnosis code distribution (low entropy = consistent coding)Contextual ConcordanceProportion of diagnosis codes assigned in clinically appropriate settings (specialist, inpatient evaluation)Medication AlignmentProportion of phenotype-coded encounters with corresponding phenotype-specific medicationTrajectory StabilityBinary indicator: 1 if most common inpatient diagnosis matches most common outpatient diagnosis Table 2Signal Fidelity Index (SFI) components.ComponentDefinitionSpecificityProportion of diagnosis codes that are specific and definitive (e.g., G30.9 Alzheimer’s vs F03 unspecified dementia)Temporal ConsistencyOne minus the frequency of diagnosis code switching across sequential encountersEntropyInverse of normalized Shannon entropy of diagnosis code distribution (low entropy = consistent coding)Contextual ConcordanceProportion of diagnosis codes assigned in clinically appropriate settings (specialist, inpatient evaluation)Medication AlignmentProportion of phenotype-coded encounters with corresponding phenotype-specific medicationTrajectory StabilityBinary indicator: 1 if most common inpatient diagnosis matches most common outpatient diagnosis Signal Fidelity Index (SFI) components. The calibration formula operates on the premise that prediction reliability correlates with diagnostic fidelity:1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{y}_{\text {calibrated}} = \hat{y}_{\text {raw}} \left[ 1 + \alpha \cdot \frac{SFI_i - \bar{SFI}_{\text {ref}}}{\bar{SFI}_{\text {ref}}} \right] \end{aligned}$$\end{document}where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}_{\text {calibrated}}$$\end{document} is the adjusted probability for patient i, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{y}_{\text {raw}}$$\end{document} is the original model prediction, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SFI_i$$\end{document} is the patient-level SFI score, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{SFI}_{\text {ref}}$$\end{document} is the mean SFI in the reference (training) dataset, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} is the calibration strength parameter. This formula amplifies predictions when patient data exhibits higher-than-average fidelity (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$SFI_i> \bar{SFI}_{\text {ref}}$$\end{document}) and attenuates them when fidelity is lower. Mathematical derivation is provided in Appendix B. Alpha selection strategy For each phenotype, optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} was determined using a principled optimization approach. We conducted a grid search testing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values from 0.5 to 2.5 in increments of 0.25, using as our performance metric the geometric mean of F1-score and Recall improvements computed as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Objective}(\alpha ) = \sqrt{\Delta F1(\alpha ) \times \Delta \text {Recall}(\alpha )}$$\end{document}. The geometric mean ensures balanced improvement across both metrics, which is particularly important for underdiagnosed conditions where F1 and Recall emphasize sensitivity to true positives. The selected \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} must demonstrate statistically significant improvement over the uncalibrated baseline as assessed by a paired t-test with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p < 0.05$$\end{document}. We also evaluated adaptive \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} selection strategies that adjust calibration strength based on test dataset characteristics, including SFI distribution and variability. We found that phenotype-specific fixed \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values provide an optimal balance between performance and simplicity. Baseline calibration methods To rigorously evaluate SFI-aware calibration, we implemented five established supervised calibration methods for direct comparison. Platt Scaling (Logistic Calibration) fits a logistic regression model to map raw predictions to calibrated probabilities according to the formula \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{\text {calibrated}} = \frac{1}{1 + \exp (A \cdot \log (\frac{p}{1-p}) + B)}$$\end{document} where A and Bare learned from training data. This method is widely used due to its simplicity and computational efficiency19. Isotonic Regression is a non-parametric method that learns a monotonic mapping from raw to calibrated probabilities using piecewise-constant isotonic regression, given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{\text {calibrated}} = \text {isotonic}(p_{\text {raw}})$$\end{document}. This approach is more flexible than Platt scaling but prone to overfitting with limited training data20. Beta Calibration is a three-parameter extension of Platt scaling accommodating non-symmetric miscalibration through the equation \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{\text {calibrated}} = \frac{1}{1 + \exp (A \cdot \log (\frac{p}{1-p}) + B \cdot [\log (\frac{p}{1-p})]^2 + C)}$$\end{document}21. Temperature Scaling is a single-parameter method dividing logits by a learned temperature T according to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{\text {calibrated}} = \frac{1}{1 + \exp (-\frac{\text {logit}(p)}{T})}$$\end{document}. This approach is widely used in deep learning due to its simplicity and effectiveness22. Histogram Binning divides prediction space into bins and assigns each bin’s empirical frequency using \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P_{\text {calibrated}} = \frac{\text {number of positive samples in bin}}{\text {total number of samples in bin}}$$\end{document}23. All baseline methods require labeled training data and were trained using the reference dataset’s training split (1,000 labeled patients). They were then applied to unlabeled test datasets–matching a realistic deployment scenario where some labeled data exists for calibration training, but the target domains have no labels. Robust error handling was implemented to prevent numerical instabilities through probability clipping, convergence checks, and fallback to uncalibrated predictions in the event of fitting failure. Experimental design For each phenotype, we conducted 10 independent simulation batches. Within each batch b, we first generated a reference dataset containing 2,000 patients with known outcomes, split into training (1,000) and testing (1,000) subsets. We then trained a reference model using a random forest classifier (with 500 trees) on the training subset, using age and race as predictors. Next, we generated 50–70 testing datasets (each comprising 1,000 patients, with phenotypes dependent on the dataset) with varying demographics and data quality. Finally, we applied calibration through uncalibrated predictions, all five baseline calibration methods (trained on reference training data), and SFI-aware calibration at multiple \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values (0.5 to 2.5). This design yielded comprehensive method comparisons across 3,550 testing datasets spanning diverse clinical conditions and data quality scenarios. Performance evaluation metrics Discrimination We assessed overall discriminative ability using the Area Under ROC Curve (AUC), computed F1-Score as the harmonic mean of Precision and Recall, measured Recall (Sensitivity) as the proportion of true cases correctly identified, and evaluated Precision (PPV) as the proportion of positive predictions that are correct. Calibration The Brier Score quantifies mean squared error of probabilistic predictions as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Brier} = \frac{1}{N}\sum _{i=1}^{N}(p_i - y_i)^2$$\end{document}. Brier Score Decomposition24 separates this into three components: Reliability measuring calibration error as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {REL} = \frac{1}{N}\sum _{k=1}^{K} n_k(\bar{p}_k - \bar{y}_k)^2$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{p}_k$$\end{document} represents mean predicted probability and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{y}_k$$\end{document} denotes observed frequency in bin k, Resolution capturing discrimination as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {RES} = \frac{1}{N}\sum _{k=1}^{K} n_k(\bar{y}_k - \bar{y})^2$$\end{document} where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{y}$$\end{document} denotes overall base rate, and Uncertainty reflecting inherent randomness as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {UNC} = \bar{y}(1-\bar{y})$$\end{document}, with the relationship \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Brier} = \text {REL} - \text {RES} + \text {UNC}$$\end{document}. Expected Calibration Error (ECE) computes the weighted average of absolute calibration errors across probability bins as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {ECE} = \sum _{k=1}^{K} \frac{n_k}{N}|\bar{p}_k - \bar{y}_k|$$\end{document}. Brier decomposition enables disentangling whether performance changes are due to improved calibration through reduced REL, improved discrimination through increased RES, or both mechanisms simultaneously. Statistical analysis Performance was aggregated at the batch level to account for the hierarchical structure of the data. For each phenotype, metric m, and batch b, we calculated mean calibrated performance across testing datasets as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{P}_{m,\text {method}}^{(b)}$$\end{document} and mean uncalibrated performance as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{P}_{m,\text {raw}}^{(b)}$$\end{document}. This yielded 50 batch-level summary values per phenotype per metric, which served as the basis for all statistical comparisons. Primary comparisons For each phenotype and method pair, we conducted paired t-tests using 50 batch-level means to assess three key comparisons. First, we tested SFI-aware versus uncalibrated predictions to determine whether SFI calibration improves performance. Second, we compared SFI-aware against each baseline method to evaluate whether SFI calibration outperforms traditional approaches. Third, we examined baselines versus uncalibrated predictions to assess whether traditional calibration methods help or harm performance. Cohen’s d quantified practical significance according to the formula \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d = \frac{\bar{x}_1 - \bar{x}_2}{s_{\text {pooled}}}$$\end{document}, where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$s_{\text {pooled}} = \sqrt{\frac{s_1^2 + s_2^2}{2}}$$\end{document}. We interpreted effect sizes as negligible when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|d| < 0.2$$\end{document}, small when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.2 \le |d| < 0.5$$\end{document}, medium when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.5 \le |d| < 0.8$$\end{document}, and large when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|d| \ge 0.8$$\end{document}. Given multiple comparisons spanning 6 phenotypes, 5 baseline methods, and 4 metrics for a total of 120 primary tests, we report both uncorrected and Bonferroni-corrected p-values with the adjusted threshold \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 0.05 / 120 = 0.000417$$\end{document}. Results remained highly significant under correction. SFI component independence assessment To assess potential redundancy among the six SFI components and validate the equal-weighting scheme, we conducted a comprehensive cross-phenotype correlation analysis. For each phenotype, we generated 10 independent batches of 50 test datasets (1,000 patients per dataset), yielding 500,000 observations per phenotype and 3,000,000 observations total across all six phenotypes. We extracted all six SFI component values (specificity, entropy, medication alignment, temporal consistency, trajectory stability, and contextual concordance) for each patient and computed pairwise Pearson correlation coefficients for all 15 unique component pairs within each phenotype. To assess cross-phenotype consistency, we calculated the mean correlations across phenotypes and evaluated the range (from minimum to maximum) for each component pair. Correlations were categorized as strong (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|r| \ge 0.7$$\end{document}), moderate (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.4 \le |r| < 0.7$$\end{document}), or weak (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|r| < 0.4$$\end{document}) following established guidelines. We computed descriptive statistics (mean, standard deviation, median, and interquartile range) for each component across all phenotypes to characterize typical value distributions and variability. Given the large sample sizes (500,000 observations per phenotype), all correlations with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|r|> 0.01$$\end{document} achieved statistical significance (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p < 0.001$$\end{document}); therefore, we focused our interpretation on effect sizes (correlation magnitudes) rather than p-values. For components exhibiting strong correlations (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|r|> 0.7$$\end{document}), we calculated the proportion of unique variance each component contributed using the complement of the average squared correlation with other components: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {Unique Variance} = 1 - \overline{R^2}$$\end{document}. This analysis informed our recommendation to retain or remove specific components from the composite SFI score. All analyses were performed in R (version 4.3.0). We offer a new R Package SFI. Complete code and phenotype parameter specifications are available at: https://github.com/clai-group/SFI Results We evaluated SFI-aware calibration across six clinically distinct phenotypes, generating 3,550 independent testing datasets. Each phenotype was assessed using uncalibrated predictions, SFI-aware calibration at phenotype-specific optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}, and five established baseline calibration methods (Platt scaling, isotonic regression, beta calibration, temperature scaling, histogram binning). All calibration methods were trained on labeled reference data (1,000 patients) and applied to testing datasets with varying demographic compositions and data quality characteristics. Complete simulation parameters for each phenotype are shown in Supplementary Table S1. Phenotype-specific optimal alpha values Alpha selection via geometric mean optimization of F1-score and Recall improvements yielded phenotype-specific optimal values presented in Table 3. Notably, optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} strongly correlated with phenotype complexity and expected diagnostic signal decay, ranging from 0.5 for fibromyalgia to 2.5 for adult ADHD, dementia, and geriatric bipolar disorder.Table 3Optimal Alpha, Recall, and Geometric Mean Across Phenotypes.PhenotypeOptimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}Optimal recallOptimal geometric meanAdult ADHD2.50.6000.630Dementia2.50.4050.465Fibromyalgia0.50.5590.603Geriatric bipolar2.50.6550.699Hypertension1.00.9150.913Type 2 diabetes2.250.8340.859 Table 3Optimal Alpha, Recall, and Geometric Mean Across Phenotypes.PhenotypeOptimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}Optimal recallOptimal geometric meanAdult ADHD2.50.6000.630Dementia2.50.4050.465Fibromyalgia0.50.5590.603Geriatric bipolar2.50.6550.699Hypertension1.00.9150.913Type 2 diabetes2.250.8340.859 Optimal Alpha, Recall, and Geometric Mean Across Phenotypes. Figure 1 demonstrates smooth, interpretable performance curves across \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values for all phenotypes. Complex phenotypes (dementia, geriatric bipolar, adult ADHD) showed monotonic or near-monotonic performance improvement with increasing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}, with optima at 2.5. Performance plateaued at high \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} but did not degrade, indicating robustness to over-calibration. Well-diagnosed phenotypes (hypertension, type 2 diabetes) exhibited inverted-U performance curves with optima at intermediate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values (1.0–2.25). Performance degraded with excessive \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} (greater than 2.0), confirming that over-calibration harms performance when baseline diagnostic quality is high. Fibromyalgia exhibited a unique pattern: optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 0.5$$\end{document} with performance degrading at higher values, suggesting that the diagnostic patterns of this phenotype are highly sensitive to calibration strength. This phenotype-dependent alpha sensitivity underscores the importance of empirical optimization for each clinical condition. However, all phenotypes showed positive performance improvement across a range of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values (typically 0.5–2.0), indicating that SFI-aware calibration is robust to moderate misspecification of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}.Fig. 1Alpha sensitivity analysis across all six phenotypes. The geometric mean of Recall shows smooth performance curves with clear optima. Complex phenotypes (dementia, geriatric bipolar, adult ADHD) show monotonic improvement with increasing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}, while well-diagnosed conditions (hypertension, type 2 diabetes) exhibit inverted-U patterns with optimal performance at intermediate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values. Alpha sensitivity analysis across all six phenotypes. The geometric mean of Recall shows smooth performance curves with clear optima. Complex phenotypes (dementia, geriatric bipolar, adult ADHD) show monotonic improvement with increasing \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}, while well-diagnosed conditions (hypertension, type 2 diabetes) exhibit inverted-U patterns with optimal performance at intermediate \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values. Table 4 presents absolute performance for uncalibrated predictions, SFI-aware calibration (at phenotype-specific optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document}), and the two most commonly used baseline methods (Platt scaling and isotonic regression). Complete results for all five baseline methods are provided in Supplementary Table S2. High diagnostic signal decay Dementia showed substantial relative improvement in F1-score (+34.9%), AUC (+7.3%), and Brier reduction (–21.8%). Optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 2.5$$\end{document} reflected high diagnostic heterogeneity. Baseline methods provided modest improvement (Platt +3.8% F1) or severe degradation (Isotonic –88.6% F1). Geriatric Bipolar demonstrated exceptional SFI-aware calibration performance (F1 = 0.746, AUC = 0.869) with +28.4% F1 improvement. High optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 2.5$$\end{document} indicated substantial diagnostic variability in psychiatric coding for older adults. Achieved lowest Brier score (0.084) and ECE (0.085) among all phenotypes. Fibromyalgia exhibited modest improvements (+4.2% F1) due to already-reasonable uncalibrated baseline (F1 = 0.625). A low optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 0.5$$\end{document} suggests that excessive calibration could harm performance. SFI-aware and Platt scaling performed essentially equivalently on F1 (negligible difference, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p=0.856$$\end{document}), but SFI-aware showed superior AUC (+5.2% vs 0%) and Brier score (–14.8% vs +5.4%). Adult ADHD showed substantial improvement (+31.6% F1, +4.7% AUC). Notable finding: Platt scaling improved F1 performance (+4.2%), but SFI-aware offered a much larger gain, underscoring that traditional calibration methods have inconsistent performance while SFI-aware calibration provides robust benefits. Low diagnostic signal decay Type 2 Diabetes showed substantial benefit despite being well-diagnosed (greater than 90% diagnosis rate), with improvements of +13.3% F1, +25.0% AUC, and –54.7% Brier. Optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 2.25$$\end{document} indicated moderate diagnostic variability likely due to variation in documentation completeness (e.g., unspecified vs type-specified diabetes codes). Achieved highest performance across all phenotypes (AUC = 0.960, F1 = 0.886). Hypertension showed remarkable improvement (+8.0% F1, +40.1% AUC, –52.6% Brier) despite near-universal diagnosis rates (greater than 95%). An optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 1.0$$\end{document} suggests that even highly standardized conditions exhibit meaningful diagnostic signal variability. Achieved strong absolute performance (AUC = 0.939, F1 = 0.910). The finding that SFI-aware calibration substantially improves performance even for well-diagnosed conditions suggests that diagnostic signal decay affects all EHR-based phenotyping, not just complex or underdiagnosed conditions. ECE analysis ECE analysis revealed nuanced patterns across phenotypes. SFI-aware calibration maintained competitive ECE performance while substantially improving Brier scores and discrimination metrics. Notably, ECE decreased for several phenotypes–indicating improved bin-level calibration–(hypertension: –45.2%, geriatric bipolar: –22.7%, type 2 diabetes: –15.4%, dementia: –14.2%, fibromyalgia: –3.5%), while adult ADHD showed a modest ECE increase (+7.7%). These ECE changes, coupled with significant gains in Brier and AUC scores, suggest that SFI-aware calibration enhances both global calibration and discrimination. The Brier decomposition confirms this interpretation: reduced reliability (improved global calibration) and increased resolution (improved discrimination) drive overall performance gains. This pattern is clinically acceptable, as the primary goal is accurate risk stratification across the full probability range, rather than perfect calibration within arbitrary bins. Future work should explore refined strategies that balance global and bin-level calibration quality.Table 4Absolute Performance Across Calibration Methods and Phenotypes.PhenotypeMethodAUCF1BrierECEAdult ADHDUncalibrated0.772 (0.764, 0.781)0.503 (0.485, 0.522)0.135 (0.130, 0.139)0.104 (0.100, 0.108)Platt Scaling0.772 (0.764, 0.781)0.524 (0.502, 0.545)0.142 (0.138, 0.147)0.096 (0.092, 0.100)Isotonic Regression0.554 (0.549, 0.560)0.044 (0.041, 0.048)0.201 (0.193, 0.208)0.160 (0.153, 0.167)SFI-Aware0.808 (0.799, 0.817)0.662 (0.643, 0.681)0.110 (0.105, 0.116)0.112 (0.107, 0.118)DementiaUncalibrated0.723 (0.714, 0.731)0.395 (0.377, 0.413)0.179 (0.174, 0.185)0.169 (0.163, 0.175)Platt Scaling0.723 (0.714, 0.731)0.410 (0.389, 0.431)0.166 (0.162, 0.170)0.132 (0.128, 0.136)Isotonic Regression0.512 (0.505, 0.519)0.045 (0.042, 0.048)0.219 (0.212, 0.226)0.151 (0.143, 0.159)SFI-Aware0.776 (0.766, 0.786)0.533 (0.512, 0.554)0.140 (0.136, 0.144)0.145 (0.140, 0.149)FibromyalgiaUncalibrated0.786 (0.779, 0.793)0.625 (0.609, 0.641)0.149 (0.145, 0.154)0.113 (0.109, 0.116)Platt Scaling0.786 (0.779, 0.793)0.643 (0.627, 0.659)0.157 (0.154, 0.161)0.119 (0.115, 0.124)Isotonic Regression0.556 (0.549, 0.562)0.113 (0.106, 0.120)0.230 (0.225, 0.235)0.194 (0.187, 0.201)SFI-Aware0.827 (0.820, 0.835)0.651 (0.635, 0.668)0.127 (0.123, 0.131)0.109 (0.105, 0.112)Geriatric BipolarUncalibrated0.801 (0.794, 0.808)0.581 (0.565, 0.597)0.140 (0.136, 0.144)0.110 (0.107, 0.114)Platt Scaling0.801 (0.794, 0.808)0.588 (0.571, 0.605)0.139 (0.136, 0.143)0.091 (0.089, 0.094)Isotonic Regression0.556 (0.549, 0.563)0.071 (0.066, 0.075)0.211 (0.205, 0.218)0.167 (0.162, 0.172)SFI-Aware0.869 (0.862, 0.877)0.746 (0.730, 0.762)0.084 (0.080, 0.087)0.085 (0.081, 0.089)HypertensionUncalibrated0.670 (0.662, 0.678)0.843 (0.839, 0.846)0.196 (0.195, 0.197)0.155 (0.153, 0.157)Platt Scaling0.670 (0.662, 0.678)0.826 (0.821, 0.830)0.207 (0.206, 0.208)0.165 (0.161, 0.169)Isotonic Regression0.530 (0.526, 0.535)0.800 (0.794, 0.806)0.225 (0.222, 0.228)0.120 (0.116, 0.125)SFI-Aware0.939 (0.936, 0.942)0.910 (0.906, 0.914)0.093 (0.090, 0.096)0.085 (0.081, 0.090)Type 2 DiabetesUncalibrated0.768 (0.760, 0.776)0.782 (0.774, 0.791)0.170 (0.168, 0.172)0.104 (0.102, 0.107)Platt Scaling0.768 (0.760, 0.776)0.787 (0.779, 0.795)0.165 (0.164, 0.167)0.087 (0.084, 0.089)Isotonic Regression0.554 (0.549, 0.558)0.429 (0.419, 0.438)0.263 (0.261, 0.264)0.189 (0.183, 0.195)SFI-Aware0.960 (0.956, 0.965)0.886 (0.878, 0.893)0.077 (0.073, 0.081)0.088 (0.082, 0.093) Table 4Absolute Performance Across Calibration Methods and Phenotypes.PhenotypeMethodAUCF1BrierECEAdult ADHDUncalibrated0.772 (0.764, 0.781)0.503 (0.485, 0.522)0.135 (0.130, 0.139)0.104 (0.100, 0.108)Platt Scaling0.772 (0.764, 0.781)0.524 (0.502, 0.545)0.142 (0.138, 0.147)0.096 (0.092, 0.100)Isotonic Regression0.554 (0.549, 0.560)0.044 (0.041, 0.048)0.201 (0.193, 0.208)0.160 (0.153, 0.167)SFI-Aware0.808 (0.799, 0.817)0.662 (0.643, 0.681)0.110 (0.105, 0.116)0.112 (0.107, 0.118)DementiaUncalibrated0.723 (0.714, 0.731)0.395 (0.377, 0.413)0.179 (0.174, 0.185)0.169 (0.163, 0.175)Platt Scaling0.723 (0.714, 0.731)0.410 (0.389, 0.431)0.166 (0.162, 0.170)0.132 (0.128, 0.136)Isotonic Regression0.512 (0.505, 0.519)0.045 (0.042, 0.048)0.219 (0.212, 0.226)0.151 (0.143, 0.159)SFI-Aware0.776 (0.766, 0.786)0.533 (0.512, 0.554)0.140 (0.136, 0.144)0.145 (0.140, 0.149)FibromyalgiaUncalibrated0.786 (0.779, 0.793)0.625 (0.609, 0.641)0.149 (0.145, 0.154)0.113 (0.109, 0.116)Platt Scaling0.786 (0.779, 0.793)0.643 (0.627, 0.659)0.157 (0.154, 0.161)0.119 (0.115, 0.124)Isotonic Regression0.556 (0.549, 0.562)0.113 (0.106, 0.120)0.230 (0.225, 0.235)0.194 (0.187, 0.201)SFI-Aware0.827 (0.820, 0.835)0.651 (0.635, 0.668)0.127 (0.123, 0.131)0.109 (0.105, 0.112)Geriatric BipolarUncalibrated0.801 (0.794, 0.808)0.581 (0.565, 0.597)0.140 (0.136, 0.144)0.110 (0.107, 0.114)Platt Scaling0.801 (0.794, 0.808)0.588 (0.571, 0.605)0.139 (0.136, 0.143)0.091 (0.089, 0.094)Isotonic Regression0.556 (0.549, 0.563)0.071 (0.066, 0.075)0.211 (0.205, 0.218)0.167 (0.162, 0.172)SFI-Aware0.869 (0.862, 0.877)0.746 (0.730, 0.762)0.084 (0.080, 0.087)0.085 (0.081, 0.089)HypertensionUncalibrated0.670 (0.662, 0.678)0.843 (0.839, 0.846)0.196 (0.195, 0.197)0.155 (0.153, 0.157)Platt Scaling0.670 (0.662, 0.678)0.826 (0.821, 0.830)0.207 (0.206, 0.208)0.165 (0.161, 0.169)Isotonic Regression0.530 (0.526, 0.535)0.800 (0.794, 0.806)0.225 (0.222, 0.228)0.120 (0.116, 0.125)SFI-Aware0.939 (0.936, 0.942)0.910 (0.906, 0.914)0.093 (0.090, 0.096)0.085 (0.081, 0.090)Type 2 DiabetesUncalibrated0.768 (0.760, 0.776)0.782 (0.774, 0.791)0.170 (0.168, 0.172)0.104 (0.102, 0.107)Platt Scaling0.768 (0.760, 0.776)0.787 (0.779, 0.795)0.165 (0.164, 0.167)0.087 (0.084, 0.089)Isotonic Regression0.554 (0.549, 0.558)0.429 (0.419, 0.438)0.263 (0.261, 0.264)0.189 (0.183, 0.195)SFI-Aware0.960 (0.956, 0.965)0.886 (0.878, 0.893)0.077 (0.073, 0.081)0.088 (0.082, 0.093) Absolute Performance Across Calibration Methods and Phenotypes. Across 3,550 datasets, no baseline calibration methods failed numerically, but functional failures were widespread. Isotonic regression collapsed AUC to near-chance levels (0.51–0.56) and severely reduced F1 scores (–91.3% for adult ADHD, –87.8% for geriatric bipolar, –45.1% for type 2 diabetes, –5.1% for hypertension). Platt scaling showed inconsistent effects, improving dementia (+3.8% F1), slightly improving type 2 diabetes (+0.6%), and reducing hypertension (–2.0%). Other methods (beta calibration, temperature scaling, histogram binning) showed similarly unstable behavior. In contrast, SFI-aware calibration never degraded performance on any phenotype or metric, demonstrating substantially greater robustness. Statistical significance of method comparisons Table 5 presents statistical comparisons between SFI-aware calibration and all competing methods. Across all phenotypes, SFI-aware calibration significantly outperformed both uncalibrated predictions and all baseline methods (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p < 0.001$$\end{document}, surviving Bonferroni correction for 120 multiple comparisons). Effect sizes were consistently large (Cohen’s \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d> 1.1$$\end{document}) for complex phenotypes and moderate to large (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d = 1.7$$\end{document}–2.4) for well-diagnosed conditions.Table 5Statistical Significance: SFI-Aware vs. Competing Methods (F1-Score).PhenotypeComparisonMean Diff95% CICohen’s dp-valueDementiaSFI vs. Uncalibrated+0.171[0.163, 0.180]1.809<0.001SFI vs. Platt+0.183[0.164, 0.203]1.122<0.001SFI vs. Isotonic+0.570[0.550, 0.590]3.203<0.001FibromyalgiaSFI vs. Uncalibrated+0.029[0.025, 0.033]0.603<0.001SFI vs. Platt−0.000[−0.005, 0.004]−0.0090.856SFI vs. Isotonic+0.560[0.546, 0.575]3.473<0.001Geriatric BipolarSFI vs. Uncalibrated+0.178[0.171, 0.186]2.174<0.001SFI vs. Platt+0.167[0.156, 0.178]1.347<0.001SFI vs. Isotonic+0.709[0.695, 0.722]5.002<0.001Adult ADHDSFI vs. Uncalibrated+0.184[0.171, 0.197]1.291<0.001SFI vs. Platt+0.147[0.126, 0.169]0.749<0.001SFI vs. Isotonic+0.662[0.641, 0.683]4.291<0.001Type 2 DiabetesSFI vs. Uncalibrated+0.103[0.098, 0.108]1.826<0.001SFI vs. Platt+0.098[0.093, 0.103]1.721<0.001SFI vs. Isotonic+0.457[0.447, 0.467]4.090<0.001HypertensionSFI vs. Uncalibrated+0.068[0.063, 0.070]2.379<0.001SFI vs. Platt+0.085[0.081, 0.088]2.293<0.001SFI vs. Isotonic+0.110[0.106, 0.114]2.413<0.001 Table 5Statistical Significance: SFI-Aware vs. Competing Methods (F1-Score).PhenotypeComparisonMean Diff95% CICohen’s dp-valueDementiaSFI vs. Uncalibrated+0.171[0.163, 0.180]1.809<0.001SFI vs. Platt+0.183[0.164, 0.203]1.122<0.001SFI vs. Isotonic+0.570[0.550, 0.590]3.203<0.001FibromyalgiaSFI vs. Uncalibrated+0.029[0.025, 0.033]0.603<0.001SFI vs. Platt−0.000[−0.005, 0.004]−0.0090.856SFI vs. Isotonic+0.560[0.546, 0.575]3.473<0.001Geriatric BipolarSFI vs. Uncalibrated+0.178[0.171, 0.186]2.174<0.001SFI vs. Platt+0.167[0.156, 0.178]1.347<0.001SFI vs. Isotonic+0.709[0.695, 0.722]5.002<0.001Adult ADHDSFI vs. Uncalibrated+0.184[0.171, 0.197]1.291<0.001SFI vs. Platt+0.147[0.126, 0.169]0.749<0.001SFI vs. Isotonic+0.662[0.641, 0.683]4.291<0.001Type 2 DiabetesSFI vs. Uncalibrated+0.103[0.098, 0.108]1.826<0.001SFI vs. Platt+0.098[0.093, 0.103]1.721<0.001SFI vs. Isotonic+0.457[0.447, 0.467]4.090<0.001HypertensionSFI vs. Uncalibrated+0.068[0.063, 0.070]2.379<0.001SFI vs. Platt+0.085[0.081, 0.088]2.293<0.001SFI vs. Isotonic+0.110[0.106, 0.114]2.413<0.001 Statistical Significance: SFI-Aware vs. Competing Methods (F1-Score). Notably, the comparison “SFI vs. Platt” for fibromyalgia showed essentially equivalent performance (mean difference near zero, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p = 0.856$$\end{document}), with a negligible effect size (Cohen’s \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d = -0.009$$\end{document}). This indicates that for phenotypes with already well-calibrated baselines and low diagnostic heterogeneity, SFI-aware calibration maintains performance without degradation. Both methods substantially outperformed isotonic regression. This underscores the importance of considering both statistical significance and practical effect sizes when interpreting results. Discrimination performance across methods Table 6Brier Score Decomposition: SFI-Aware Calibration (with 95% CI).PhenotypeMethodBrierReliabilityResolution Adult ADHDUncalibrated0.135 (0.130, 0.139)0.017 (0.016, 0.019)0.055 (0.052, 0.058)SFI-Aware0.110 (0.105, 0.116)0.028 (0.027, 0.029)0.089 (0.085, 0.094) DementiaUncalibrated0.179 (0.174, 0.185)0.038 (0.036, 0.040)0.043 (0.040, 0.045)SFI-Aware0.140 (0.136, 0.144)0.034 (0.033, 0.035)0.079 (0.075, 0.083) FibromyalgiaUncalibrated0.149 (0.145, 0.154)0.019 (0.018, 0.020)0.063 (0.060, 0.065)SFI-Aware0.127 (0.123, 0.131)0.019 (0.018, 0.020)0.086 (0.082, 0.089) Geriatric BipolarUncalibrated0.140 (0.136, 0.144)0.018 (0.017, 0.019)0.063 (0.060, 0.066)SFI-Aware0.084 (0.080, 0.087)0.016 (0.015, 0.017)0.118 (0.113, 0.122) HypertensionUncalibrated0.196 (0.195, 0.197)0.028 (0.027, 0.029)0.040 (0.038, 0.043)SFI-Aware0.093 (0.090, 0.096)0.020 (0.019, 0.020)0.135 (0.133, 0.137) Type 2 DiabetesUncalibrated0.170 (0.168, 0.172)0.016 (0.016, 0.017)0.070 (0.067, 0.072)SFI-Aware0.077 (0.073, 0.081)0.025 (0.022, 0.027)0.169 (0.166, 0.172) Table 6Brier Score Decomposition: SFI-Aware Calibration (with 95% CI).PhenotypeMethodBrierReliabilityResolution Adult ADHDUncalibrated0.135 (0.130, 0.139)0.017 (0.016, 0.019)0.055 (0.052, 0.058)SFI-Aware0.110 (0.105, 0.116)0.028 (0.027, 0.029)0.089 (0.085, 0.094) DementiaUncalibrated0.179 (0.174, 0.185)0.038 (0.036, 0.040)0.043 (0.040, 0.045)SFI-Aware0.140 (0.136, 0.144)0.034 (0.033, 0.035)0.079 (0.075, 0.083) FibromyalgiaUncalibrated0.149 (0.145, 0.154)0.019 (0.018, 0.020)0.063 (0.060, 0.065)SFI-Aware0.127 (0.123, 0.131)0.019 (0.018, 0.020)0.086 (0.082, 0.089) Geriatric BipolarUncalibrated0.140 (0.136, 0.144)0.018 (0.017, 0.019)0.063 (0.060, 0.066)SFI-Aware0.084 (0.080, 0.087)0.016 (0.015, 0.017)0.118 (0.113, 0.122) HypertensionUncalibrated0.196 (0.195, 0.197)0.028 (0.027, 0.029)0.040 (0.038, 0.043)SFI-Aware0.093 (0.090, 0.096)0.020 (0.019, 0.020)0.135 (0.133, 0.137) Type 2 DiabetesUncalibrated0.170 (0.168, 0.172)0.016 (0.016, 0.017)0.070 (0.067, 0.072)SFI-Aware0.077 (0.073, 0.081)0.025 (0.022, 0.027)0.169 (0.166, 0.172) Brier Score Decomposition: SFI-Aware Calibration (with 95% CI). Table 6 provides a complete Brier decomposition into reliability reflecting calibration, resolution reflecting discrimination, and uncertainty components. Brier decomposition reveals that SFI-aware calibration improves performance through both enhanced calibration, reflected in a reduced reliability component, and improved discrimination, reflected in an increased resolution component. Regarding reliability reduction, SFI-aware calibration reduced calibration error for dementia (−11%), geriatric bipolar (−11%), and hypertension (−29%), indicating substantially improved alignment between predicted probabilities and observed frequencies. For resolution enhancement, SFI-aware calibration simultaneously increased resolution or discrimination ability across all phenotypes, including +84% for dementia, +87% for geriatric bipolar, +35% for fibromyalgia, +62% for adult ADHD, +141% for type 2 diabetes, and +238% for hypertension, demonstrating that fidelity-aware adjustment not only calibrates probabilities but also enhances the model’s ability to separate positive from negative cases. The overall Brier improvement from these combined effects yielded 15–54% Brier score reduction across phenotypes, including −22% for dementia, −40% for geriatric bipolar, −15% for fibromyalgia, −19% for adult ADHD, −54% for type 2 diabetes, and −53% for hypertension, with the largest gains occurring in conditions exhibiting the highest baseline miscalibration. This dual mechanism of simultaneously improving calibration and discrimination distinguishes SFI-aware calibration from traditional methods, which typically improve one at the expense of the other. For instance, Platt scaling may improve calibration but reduce discrimination. Component Independence and Weighting Correlation analysis (Figure 1S) revealed one pair of highly redundant components, alongside five independent components that measure distinct data quality dimensions. Specificity and contextual concordance exhibited a near-perfect correlation (mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r = 0.958$$\end{document}, range: 0.940 to 0.971 across phenotypes), indicating these components measure highly overlapping constructs in our simulation data. While these components showed high correlation in our simulation, they measure conceptually distinct constructs with different denominators. Specificity is defined as the proportion of specific diagnosis codes among all phenotype-related codes, whereas contextual concordance is defined as the proportion of diagnosis codes documented in clinically appropriate settings among all patient encounters. The high correlation occurs because both denominators (total phenotype codes and total encounters) are themselves highly correlated in typical clinical documentation patterns, where patients with more encounters tend to accumulate more phenotype-specific codes. Critically, these denominators can diverge substantially for certain phenotypes or patient populations that differ from our simulation parameters. Phenotypes with very sparse coding patterns, where patients have many encounters but few phenotype-specific codes documented (such as early-stage chronic conditions or episodic acute conditions), would show discordance between these components. Similarly, phenotypes where nearly all codes are specific but documented across unusually diverse care settings (such as conditions requiring both emergency and specialty care) would show high specificity but variable contextual concordance. Populations with fragmented care patterns may have high encounter counts relative to phenotype-specific coding intensity, again creating divergence between these measures. Retaining both components provides no computational harm, given the minimal added cost (one additional ratio calculation per patient) while preserving flexibility and generalizability for phenotypes and populations where the correlation observed in our simulation may not hold. This design choice prioritizes robustness across diverse real-world clinical contexts over optimization for our specific simulation parameters. The additional information captured by maintaining both denominators (phenotype codes versus total encounters) may prove valuable in healthcare settings with documentation patterns that differ substantially from those we simulated. Two component pairs showed moderate correlations: specificity and medication alignment (mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r = 0.696$$\end{document}, range: 0.641–0.745), and medication alignment and contextual concordance (mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r = 0.674$$\end{document}, range: 0.614–0.736). These moderate correlations are clinically interpretable – i.e., patients with specific diagnoses tend to receive appropriate medications, and appropriate medications co-occur with appropriate clinical settings – but each component retains substantial unique variance (48% for medication alignment and 40% for specificity). The moderate correlation between specificity and medication alignment represents conceptually complementary rather than redundant information. Specificity captures diagnostic code quality, while medication alignment captures treatment concordance, providing independent evidence of diagnostic reliability from distinct data sources (codes versus medications). The remaining component pairs exhibited weak correlations (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|r| < 0.4$$\end{document}), with the majority showing near-zero relationships. Entropy correlated weakly with temporal consistency (mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r = 0.386$$\end{document}), reflecting the intuitive relationship between diagnostic code diversity and consistency; however, this correlation explained only 15% of the variance. Trajectory stability demonstrated near-independence from all other components (all \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|r| < 0.03$$\end{document}), contributing greater than 95% unique variance despite its binary formulation. Similarly, temporal consistency showed weak correlations with most components (accounting for 77% of the unique variance), and entropy remained largely independent (accounting for 82% of the unique variance). Cross-phenotype consistency was high for most correlation pairs, with 83% showing ranges less than 0.15, supporting the generalizability of component relationships across diverse clinical contexts. Discussion This study presented and evaluated SFI-aware calibration across six clinically diverse phenotypes and 3,550 synthetic EHR datasets, demonstrating that diagnostic signal decay1 – i.e., systematic variability in diagnostic data quality across healthcare settings – can be quantified at the patient level and addressed through targeted calibration. Results showed consistent performance improvements over both uncalibrated predictions and established baseline methods across all evaluated phenotypes. While distributional shift and domain adaptation are well-studied phenomena in machine learning9,10, this work operationalizes diagnostic signal decay as a clinically-grounded framework rather than treating diagnostic code heterogeneity as statistical noise. Rather than applying adversarial training25 or instance reweighting26 that require continuous model retraining and labeled target data, we quantify data reliability at the patient level using structured metadata alone. The SFI provides a measurable proxy for diagnostic fidelity, enabling calibration without outcome labels in target populations and addressing a key limitation of traditional supervised calibration methods. Traditional calibration methods showed marked instability in our evaluation. Isotonic regression degraded to near-chance performance across all phenotypes, while Platt scaling demonstrated inconsistent, phenotype-dependent effects. This failure pattern reflects fundamental limitations when global mappings learned from training data encounter systematically different testing distributions. Traditional methods treat all predictions equivalently, applying uniform calibration adjustments regardless of underlying data quality. A prediction derived from a patient with specific diagnostic codes, specialist encounters, and appropriate medications receives identical calibration as one based on non-specific codes from a single primary care visit. This uniformity assumption breaks down when diagnostic signal quality varies within the target population. By computing patient-level fidelity scores, SFI-aware calibration adapts to heterogeneous data characteristics, amplifying high-fidelity predictions while attenuating low-fidelity ones without requiring outcome labels. Brier decomposition revealed that SFI-aware calibration operates through dual mechanisms: enhanced calibration and improved discrimination. Traditional calibration methods typically face a calibration-discrimination tradeoff, where improving one aspect often comes at the expense of the other21. SFI-aware calibration avoids this tradeoff because the SFI functions as a feature-derived confidence estimator. High SFI scores indicate patients for whom diagnostic codes reliably reflect the true disease status, enabling the selective upweighting of predictions where the base model is most trustworthy. Conversely, low SFI scores identify patients where diagnostic codes are noisy or unreliable, prompting conservative prediction attenuation. This selective weighting effectively increases signal-to-noise ratio, improving the model’s ability to separate true positives from false positives. The modest ECE increases observed for several phenotypes merit clarification. ECE measures bin-level calibration by dividing the probability space into fixed bins and comparing predicted versus observed frequencies within each bin. In contrast, the Brier score measures global calibration across the entire probability distribution. SFI-aware calibration optimizes global calibration and discrimination, resulting in modest local bin-level calibration changes. This tradeoff is reasonable because accurate risk stratification across the full probability range takes precedence over perfect calibration within arbitrary bins. The Brier decomposition confirms this interpretation: reduced reliability (improved global calibration) and increased resolution (improved discrimination) drive overall performance gains. Multi-phenotype evaluation revealed systematic patterns. Complex, under-diagnosed phenotypes with high diagnostic signal decay (dementia, adult ADHD, geriatric bipolar disorder) showed larger relative improvements (F1 +28.4% to +34.9%) with high optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values (2.5), reflecting substantial diagnostic variability requiring stronger calibration adjustments. More established phenotypes with standardized diagnostic criteria (type 2 diabetes, hypertension) still demonstrated substantial gains (F1 +8.1% to +13.0%, AUC +24.7% to +40%), indicating that documentation heterogeneity affects all EHR-based phenotyping, not just complex conditions. Optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} values varied from 0.5 to 2.5, emphasizing the need for phenotype-specific calibration strength. However, all phenotypes showed positive improvements across wide \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} ranges (typically 0.5–2.0), indicating robustness to moderate misspecification. Component correlation analysis revealed that specificity and contextual concordance are highly correlated (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r = 0.96$$\end{document}) in our simulation. However, these components measure conceptually distinct constructs with different denominators: specificity normalizes by total phenotype codes while contextual concordance normalizes by total encounters. While correlation is high in typical documentation patterns, these denominators can diverge substantially for phenotypes with sparse coding patterns (few phenotype codes despite many encounters), fragmented care (high encounter counts relative to phenotype-specific coding), or unusual documentation characteristics. We retain both components, given minimal computational cost and the need for generalizability to diverse real-world contexts where observed correlations may not hold. Similarly, the moderate correlation between specificity and medication alignment (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r = 0.70$$\end{document}) represents complementary rather than redundant information, as specificity captures diagnostic code quality, while medication alignment captures treatment concordance, providing independent evidence from distinct data sources. The remaining component pairs exhibited weak correlations, with trajectory stability demonstrating near-independence from all other components. Computational efficiency enables large-scale deployment. SFI computation scales linearly with patient encounters. For Medicare-scale deployment (60 million beneficiaries), we expect that parallel processing on a 64-core server would require a few hours, compared to weeks or months for model retraining. After one-time \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} optimization using labeled validation data, the method can be applied to unlimited unlabeled target populations, utilizing only structured metadata. This computational profile makes SFI-aware calibration suitable for large-scale, multi-site deployment where traditional calibration approaches requiring labeled data at each site are infeasible. Fairness considerations warrant careful attention. SFI components encode diagnostic practices that may differ systematically across patient subgroups. Contextual concordance may penalize patients with limited access to specialists, even when diagnoses are accurate. Medication alignment may disadvantage populations facing cost barriers or insurance restrictions. Mitigation strategies may include audits stratified by demographic characteristics, component reweighting in heterogeneous populations with known access disparities, and subgroup-specific calibration. Rigorous fairness audits using real multi-site EHR data across diverse patient populations are essential before clinical deployment. Real-world validation is crucial for the clinical application of the SFI-aware calibration. We propose a phased validation strategy: single-site EHR validation on data with expert-adjudicated outcomes to compare simulated versus real SFI distributions and assess calibration performance; multi-site validation using national cohorts (with robust outcome adjudications) with train-on-one, test-on-many deployment; and large-scale claims validation using Medicare Fee-for-Service (60M+ beneficiaries) to assess computational scalability and performance using proxy outcomes. Limitations Several limitations warrant consideration. While our simulation incorporated realistic epidemiological parameters and clinical coding patterns, it cannot fully capture the complexity of real-world data, including missing data, documentation errors, and intricate comorbidity interactions. Real-world multi-site validation is crucial for assessing clinical translation. The SFI components were designed for ICD-coded phenotypes. Adaptation to laboratory-defined (e.g., chronic kidney disease via eGFR), imaging-defined (e.g., stroke via CT), or procedure-defined phenotypes (e.g., coronary artery disease via catheterization) would require re-engineering of the components. The extent to which these alternative phenotyping strategies can be generalized remains to be determined. Our approach selects a single optimal \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha$$\end{document} per phenotype, assuming consistent SFI-reliability relationships across deployment settings. In practice, this relationship may vary across institutions. Adaptive alpha selection, based on target population characteristics, could address this limitation; however, it introduces additional complexity. We evaluated SFI-aware calibration using random forest models with demographic predictors. Performance may differ with deep learning architectures, rich feature sets incorporating medications and laboratory results, or ensemble models. While the calibration formula is model-agnostic in principle, empirical validation across diverse model architectures is needed. All simulated data spanned a fixed 5-year period. Real-world deployment must contend with temporal drift, including evolving diagnostic criteria, new medications, and changing billing practices. Whether SFI-aware calibration maintains effectiveness across multi-year deployments with a temporal distribution shift requires longitudinal validation. Outcome labels in the simulation were generated from known epidemiological knowledge. Real-world labels involve expert disagreement, measurement error, and confounding by indication. The performance of SFI-aware calibration when trained on imperfect labels requires empirical assessment. Conclusion This study demonstrates that diagnostic signal decay–systematic variability in diagnostic data quality across healthcare settings–can be addressed through the quantification and calibration of patient-level fidelity. Evaluation across six phenotypes and 3,550 synthetic datasets demonstrated consistent performance improvements (F1 score +4.2% to +34.9%, AUC +4.6 % to +40%) over uncalibrated predictions and established baseline methods. SFI-aware calibration improved both calibration quality (reliability reduction) and discrimination ability (resolution increase), a combination rarely achieved by traditional calibration methods. The method offers practical advantages for clinical deployment. After one-time parameter optimization using labeled validation data, it can be applied to unlimited unlabeled target populations using only structured EHR metadata. Computational efficiency (2–4 hours for 60 million Medicare beneficiaries) enables large-scale implementation. The approach is particularly suitable for administrative datasets where outcome labels are unavailable. Multi-phenotype validation confirmed generalizability across clinical conditions. Benefits are extended even to well-diagnosed conditions with standardized criteria (such as type 2 diabetes and hypertension), indicating that documentation heterogeneity affects all EHR-based phenotyping. Optimal calibration strength varied by phenotype complexity (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = 0.5$$\end{document} to 2.5), emphasizing the need for condition-specific parameter selection. Future work should focus on real-world validation following our proposed four-phase strategy, extending to laboratory-defined and imaging-defined phenotypes, integrating with uncertainty quantification methods such as conformal prediction, and exploring adaptive calibration strategies that adjust to target population characteristics. The core concept of quantifying data fidelity for personalized calibration may extend beyond healthcare to other domains with heterogeneous administrative data. This work provides evidence that diagnostic signal decay, while challenging, can be systematically addressed through fidelity-aware calibration. The method offers a practical approach for maintaining prediction model performance across heterogeneous healthcare contexts, with clear implications for large-scale deployment where traditional calibration approaches are infeasible. Supplementary Information Supplementary Information. Supplementary Information.