Integrating Clinical Priorities into the Technical Validation of Machine Learning Classifiers: A Generalizable Framework Demonstrated on Opioid Misuse Screening Authors: Jacob Washton, Milan Toma Published: 2026 ABSTRACT Background/Objectives: Traditional machine learning metrics often fail to capture the clinical consequences of classification errors, particularly in high-stakes screening applications. When applying task-specific AI tools to human lives, clinicians cannot rely on simple, headline metrics and marketing materials. Impressive accuracy figures can hide dangerous algorithmic shortcuts that collapse when encountering actual patients. This study addresses the gap between statistical performance and clinical utility by evaluating classifiers for prescription opioid misuse using a clinically oriented framework. Methods: Three ensemble models, namely, Cost-Sensitive Bagged Trees (CSBT-Untuned and CSBT-Tuned) and Random Undersampling Boosting (RUSBoost), were trained on National Survey on Drug Use and Health data and assessed using composite metrics integrating clinical priorities and asymmetric error costs. Results: The results demonstrate that while CSBT-Untuned achieved the highest raw accuracy of 86.7%, it missed 61.5% of positive cases. Conversely, the threshold-optimized CSBT-Tuned model achieved enhanced minority class detection and numerically higher Clinical Discriminative Performance Scores under sensitivity-priority scenarios, though its performance remained statistically comparable to RUSBoost given overlapping confidence intervals. Learning curve analysis confirmed stable convergence for the cost-sensitive bagging approach. Conclusions: Before accepting AI tools in patient care, physicians must demand an evaluation report similarly detailed to this manuscript, utilizing this framework as guidance for what an evidence report should demonstrate prior to clinical deployment. 1. Introduction The clinical integration of artificial intelligence is accelerating, with diagnostic aids, predictive risk models, and decision support systems emerging across medical specialties at a pace that evaluation frameworks oriented toward technical validation have failed to match [1]. Central to this challenge is the recognition that evaluation methodology itself has become the critical bottleneck, i.e., the question is no longer whether machine learning models can achieve strong predictive performance, but whether the criteria used to assess that performance reflect the priorities of clinicians and the outcomes that matter to patients. 1.1. Background and Clinical Motivation The evaluation of machine learning classifiers in medical contexts presents a fundamental challenge: traditional statistical performance metrics, while mathematically rigorous, often fail to capture the clinical consequences that ultimately determine a model’s value for patient care [1,2]. This limitation is evident across diverse prediction domains. In healthcare mortality prediction, studies employing deep learning architectures such as convolutional neural networks with gated recurrent units have achieved strong performance as measured by mean absolute error, mean squared error, and coefficient of determination, yet these conventional regression metrics provide no mechanism for encoding the asymmetric clinical consequences of prediction errors [3]. Similarly, classification studies in domains ranging from acoustic gender recognition [4] to behavioral survey analysis [5] have reported high raw accuracy and F1 scores without addressing whether such metrics adequately reflect the decision-relevant properties of the prediction task. The weighted linear combination follows from multi-attribute utility theory and aligns with clinical guidelines on healthcare composite performance assessment [6], which establishes that when component attributes are mutually preferentially independent, a decision maker’s preferences can be represented as an additive value function over the individual attributes. Importantly, this disconnect between predictive sophistication and evaluation adequacy is not unique to healthcare. Recent advances in ensemble learning and hybrid architectures have demonstrated the capacity of tree-based and deep learning models to achieve strong predictive performance across diverse high-stakes forecasting applications, from agricultural yield prediction using gradient boosting with meta-heuristic optimization [7] to climate variable forecasting using hybrid CNN-ResNet-LSTM architectures for renewable energy decision support [8] and emissions prediction using deep neural network and gradient boosting frameworks [9]. In each of these domains, as in healthcare, decision makers face asymmetric consequences: underestimating crop yield affects food security planning differently than overestimation, and underestimating emissions carries different policy implications than overestimation. Yet these methodological advances have largely proceeded independently of efforts to develop decision-analytic evaluation criteria appropriate for contexts where error costs are asymmetric and stakes are high. The result is a proliferation of sophisticated predictive models evaluated using metrics that, while statistically sound, fail to capture the priorities of decision makers. This pattern is especially consequential in medical applications, where the asymmetry between missed diagnoses and false alarms can translate directly into patient harm, making healthcare the focus of the present work. The problem of predicting lifetime prescription opioid misuse using data from the National Survey on Drug Use and Health (NSDUH) is used to apply a clinically oriented evaluation protocol, designed to bridge this gap through composite metrics that combine multiple performance dimensions with adjustable weights reflecting clinical priorities. The problem of predicting lifetime prescription opioid misuse is a classification task characterized by substantial class imbalance and asymmetric error costs that render traditional accuracy metrics particularly misleading [10,11,12]. This accuracy paradox, wherein high overall accuracy obscures poor performance on the clinically critical minority class, is not unique to medical screening but is rarely addressed in machine learning evaluation practice. Studies achieving classification accuracies exceeding 99% on balanced datasets [4] demonstrate that raw accuracy can serve as a meaningful performance indicator when class distributions are symmetric and error costs are equivalent, but these conditions seldom hold in clinical contexts where the base rate of the target condition may be low and the consequences of missed cases may substantially exceed the consequences of false alarms. Beyond applying the foundational metrics, a detailed suite of methodological extensions addressing practical limitations inherent in any weighting-based evaluation approach was developed and demonstrated in this study. Building on algorithmic foundations shared with ensemble architectures deployed across non-medical domains [4,7,9], this study extends beyond classifier training to focus explicitly on evaluation methodology. Recognizing that a model’s clinical value depends on how statistical performance translates into patient outcomes under asymmetric cost structures, the present work formalizes a decision-analytic evaluation framework tailored to healthcare screening. 1.2. The Challenge of Opioid Misuse Prediction Prescription opioid misuse represents a significant public health concern, with millions of individuals affected annually and substantial associated morbidity, mortality, and healthcare costs [13,14]. Early identification of individuals with opioid misuse history enables targeted intervention, risk-stratified prescribing practices, and appropriate allocation of treatment resources [15,16,17,18]. Machine learning classifiers trained on demographic and behavioral survey data offer potential for population-level screening, yet their evaluation requires careful consideration of the clinical context in which they would be deployed [11,19,20]. However, existing machine learning studies in this domain predominantly rely on conventional metrics evaluated in isolation without explicit integration of clinical priorities. Because context-agnostic indices summarize performance across arbitrary operating boundaries or conflate majority class true negatives with minority class case detection, they fail to provide a principled mechanism for trading off sensitivity against specificity according to clinical stakes. This gap motivates our decision-analytic framework. The prediction of opioid misuse exemplifies the challenges that motivate clinically oriented evaluation. Class imbalance is pronounced: in the NSDUH sample examined here, only 11.7% of respondents report lifetime prescription opioid misuse, creating a majority-class-dominated accuracy landscape where a trivial classifier predicting all-negative would achieve nearly 90% accuracy while providing no clinical value. Error costs are asymmetric: missing an individual with opioid misuse history (false negative) may delay intervention during a critical window, while incorrectly flagging an individual without such history (false positive) imposes assessment burden but typically carries lower clinical consequence. These characteristics demand evaluation metrics sensitive to minority class detection and capable of incorporating differential error costs. 1.3. Study Objectives and Organization The primary objective of this study is to evaluate machine learning classifiers for prescription opioid misuse prediction using a clinically oriented evaluation framework. This framework is distinct from conventional machine learning evaluation, which relies on context-agnostic metrics such as accuracy, F1 score, or area under the receiver operating characteristic curve that treat all classification errors as equivalent. It also differs from generic multi-criteria decision-making (MCDM) approaches such as the Analytic Hierarchy Process or TOPSIS, which, although capable of synthesizing multiple objectives, typically lack the domain-specific parameterization required for medical contexts and do not directly encode the asymmetric clinical consequences inherent in diagnostic decision making [21]. This approach moves beyond traditional statistical metrics by integrating medical priorities and asymmetric error costs into the technical validation process through explicitly parameterized weighting schemes that embed clinical judgment directly into the evaluation structure. We demonstrate this framework through four composite metrics and a suite of methodological extensions designed to address practical deployment challenges such as weight selection, uncertainty quantification, and decision-analytic integration. Three classification approaches are compared: cost-sensitive bagged trees with default threshold (CSBT-Untuned), cost-sensitive bagged trees with optimized threshold (CSBT-Tuned), and Random Undersampling Boosting (RUSBoost). Their comparative evaluation illuminates how different clinical priorities yield different model recommendations, providing a template for practitioners to adapt to other medical classification problems. The secondary objective is to demonstrate the practical application of the methodological extensions, providing a template for comprehensive clinically oriented evaluation that practitioners can adapt to other medical classification problems. The modular nature of the framework is emphasized: not every extension is required for every application, and practitioners should select those appropriate to the specific demands of each clinical context. The framework presented here provides a principled approach to medical machine learning evaluation that respects both the rigor of statistical analysis and the complexity of clinical decision making. As physicians and end users of diagnostic tools, clinicians are directly responsible for decisions that alter patient lives and cannot outsource their clinical judgment to marketing materials. Would a practitioner trust an AI tool boasting 95% accuracy over one claiming 85%? In reality, impressive aggregate accuracy numbers frequently mask algorithmic shortcuts that collapse the moment the model encounters actual patients. However, it must be recognized that machine learning models, much like large language models, are fundamentally statistical tools. They function effectively for individuals who fit the training statistics, and they cannot be expected to serve as confirmatory instruments because specific patients will always exist whose unique clinical presentations cannot be described by aggregate data. Using these tools is essentially identifying a specific condition in the patient at hand based on nationwide statistics, concluding that because most people in a similar situation had condition A, this patient has it too, even though their unique situation could be explained by other factors. The mathematics provides structure and precision; clinical expertise provides the necessary priorities, weights, and individual nuance; together, they enable evaluation that serves the ultimate purpose of medical artificial intelligence, i.e., the improvement of patient care. 2. Materials and Methods This section details the empirical pipeline and decision-analytic evaluation protocol developed to assess machine learning classifiers under clinical priority constraints. The methodological workflow encompasses dataset construction and preprocessing, ensemble model training across three distinct classification pipelines, learning dynamics evaluation, and a multidimensional clinical validation framework incorporating cost matrices, expert swing-weight elicitation, net benefit analysis, and cost-effectiveness modeling. 2.1. Study Design and Data Source The present study employed a supervised machine learning approach to predict lifetime prescription opioid misuse using data derived from the National Survey on Drug Use and Health (NSDUH) spanning the years 2015 through 2019. The outcome variable was defined as a binary indicator representing any history of prescription pain reliever misuse, constructed from the original survey variable pnrnmrec. Based on the NSDUH codebook, respondents with codes indicating misuse within the past 30 days (code 1), misuse more than 30 days ago but within the past 12 months (code 2), or misuse more than 12 months ago (code 3) were classified as positive cases, whereas respondents coded as never having used or misused pain relievers (code 91) were classified as negative cases. Observations with ambiguous or missing outcome values were excluded from analysis. 2.2. Data Preprocessing All preprocessing operations were fitted exclusively on the training partition to prevent information leakage from validation and test sets; specifically, median imputation parameters and z-score scaling parameters (μj,σj) were calculated solely from training observations and subsequently applied to transform validation and test vectors without updating or re-fitting. Survey response codes commonly representing missing, refused, or unknown values (specifically codes 85, 94, 97, 98, and 99) were converted to missing indicators. Observations exceeding 50 percent missingness across predictor variables were removed from the training set. Variables potentially exhibiting target leakage, including alternative encodings of the outcome variable such as pnrnmyr, pnrnmevr, pnrnm30d, and pnrnm30fq, were identified and excluded prior to model development. Numeric predictors underwent median imputation followed by z-score standardization, wherein each feature xj was transformed according to(1)zj=xj−μjσj where μj and σj denote the mean and standard deviation of feature j computed from the training set. Categorical predictors were converted to dummy variables using one-hot encoding with the first category serving as the reference level. Features exhibiting zero variance in the training set were removed from all partitions. Any residual missing values following imputation were replaced with zeros. The complete preprocessing procedure, including the fitting and transformation functions, is formally specified in Algorithm A1. 2.3. Data Partitioning The preprocessed dataset was divided into three mutually exclusive subsets using stratified random sampling to preserve class proportions across partitions. The primary partition allocated 70 percent of observations to the training set, with the remaining 30 percent subsequently split equally between validation and test sets, yielding a final distribution of approximately 70 percent training, 15 percent validation, and 15 percent testing. The validation set served exclusively for hyperparameter assessment and threshold optimization, whereas the test set was reserved in strict holdout isolation for final unbiased performance evaluation, ensuring that test data played no role in feature transformation, hyperparameter selection, or decision boundary calibration. Figure 1 illustrates the data partitioning strategy employed across all three modeling pipelines. 2.4. Classification Models Three ensemble classification models were developed and evaluated: Cost-Sensitive Bagged Trees without threshold tuning (CSBT-Untuned), Cost-Sensitive Bagged Trees with threshold tuning (CSBT-Tuned), and Random Under-Sampling Boosting (RUSBoost). All models utilized decision trees as base learners within their respective ensemble frameworks. The relationship among the three modeling pipelines is depicted in Figure 2, which illustrates the shared preprocessing infrastructure and divergent model-specific components. 2.4.1. Cost-Sensitive Bagged Trees The Cost-Sensitive Bagged Trees models employed bootstrap aggregation (bagging) combined with asymmetric misclassification costs to address class imbalance. Bagging constructs an ensemble of B decision trees, each trained on a bootstrap sample drawn with replacement from the training data. For a new observation x, the ensemble prediction is obtained by aggregating individual tree predictions. In the classification setting, each tree hb(x) casts a vote, and the final predicted class is determined by(2)y^(x)=argmaxc∈{0,1}∑b=1BIhb(x)=c where I(·) denotes the indicator function, and B represents the total number of trees in the ensemble. The internal iteration structure of the bagging procedure is depicted in Figure 3. To account for the differential consequences of classification errors in the context of opioid misuse prediction, a cost matrix C was incorporated during training. The cost matrix penalizes false negatives (missed cases of misuse) more heavily than false positives, reflecting the clinical importance of identifying individuals at risk. The cost matrix was specified as(3)C=0cFPcFN0 where cFP=1 denotes the cost of a false positive, and cFN=3 denotes the cost of a false negative. During tree construction, split criteria incorporate these costs to minimize expected misclassification cost rather than simple error rate. The CSBT-Untuned model employed the default decision threshold of 0.5 for converting predicted probabilities to class labels. The CSBT-Tuned model extended this approach by optimizing the decision threshold exclusively on the validation set, guaranteeing that no information from the independent test set was utilized during calibration. For threshold tuning, predicted probabilities p^(x) generated on validation feature vectors Xval were converted to class labels according to(4)y^(x)=1ifp^(x)≥τ0otherwise The threshold optimization employed a grid search over the interval [0.05,0.95] with increments of 0.01, evaluating each candidate threshold τ against the validation set, thereby completely decoupling decision boundary calibration from both initial tree ensemble training and final test set evaluation. The optimal threshold τ* was selected as(5)τ*=argmaxτ∈{0.05,0.06,…,0.95}F1(τ) where F1(τ) denotes the F1 score achieved on the validation set at threshold τ; once determined, τ* was locked and applied directly to the unseen holdout test set without further modification. This selection criterion balances precision and recall, providing a principled approach to threshold selection that accounts for class imbalance. The complete pseudocode for the CSBT-Untuned pipeline is provided in Algorithm A2, while the threshold-tuned variant is detailed in Algorithm A3. 2.4.2. RUSBoost The RUSBoost algorithm combines random undersampling of the majority class with boosting to address class imbalance. At each boosting iteration t, the algorithm creates a balanced training subset by randomly undersampling the majority class to match the minority class size [22,23]. A weak learner (decision tree) is then trained on this balanced subset, and instance weights are updated based on classification performance. The internal iteration structure is depicted in Figure 4. The boosting procedure maintains a weight distribution Dt(i) over training instances. After training weak learner ht, the weighted error is computed as(6)ϵt=∑i:ht(xi)≠yiDt(i) The learner weight αt is then calculated based on the error rate, and instance weights are updated to emphasize misclassified observations in subsequent iterations. Specifically, the learner weight is computed as(7)αt=η·12ln1−ϵtϵt where η denotes the learning rate parameter that controls the contribution of each weak learner to the ensemble. The final ensemble prediction combines all weak learners according to(8)H(x)=sign∑t=1Tαtht(x) where T denotes the total number of boosting iterations. The workflow for the RUSBoost pipeline is illustrated in Figure 4, with complete pseudocode provided in Algorithm A4. 2.4.3. Unified Model Workflows The complete workflow for all three pipelines is presented in Figure 5. The diagram illustrates the shared preprocessing infrastructure at the top, the three parallel model-specific training branches in the middle, and the shared evaluation procedures at the bottom. The CSBT-Untuned pipeline (a) proceeds directly from training to evaluation, the CSBT-Tuned pipeline (b) includes an additional threshold optimization step, and the RUSBoost pipeline (c) employs a different training algorithm but shares all other components with the CSBT models. Detailed pseudocode implementations for all three pipelines, including shared preprocessing functions and evaluation metrics, are provided in Appendix A. 2.5. Hyperparameter Configuration The hyperparameters for each model were configured as summarized in Table 1 based on a grid search optimization procedure evaluated on the validation set partition. Candidate search ranges evaluated during hyperparameter selection included ensemble size B∈{50,100,150,200}, maximum tree splits S∈{20,50,100,200}, false negative cost ratios cFN∈{1,2,3,5}, learning rate η∈{0.01,0.05,0.1,0.2}, and decision threshold search grid τ∈[0.05,0.95] at 0.01 increments. Model training stopping criteria were defined by fixed ensemble iteration limits (B=150), with convergence confirmed via learning curve inspection to ensure stability without premature termination or overfitting. Both CSBT models utilized identical ensemble and cost parameters, differing only in the threshold optimization procedure. The RUSBoost model employed a learning rate parameter to control the contribution of each weak learner to the ensemble. All competing models were subjected to an identical systematic grid search optimization protocol across equivalent hyperparameter search space dimensions on the validation partition, ensuring a fair comparative evaluation baseline. The cost matrix configuration for the CSBT models assigned a false negative cost three times greater than the false positive cost, reflecting the priority of identifying individuals with opioid misuse history. The threshold optimization for CSBT-Tuned searched over 91 candidate thresholds from 0.05 to 0.95 in increments of 0.01, selecting the threshold that maximized the F1 score on the validation set. Table 2 presents the preprocessing and data handling parameters common to all three pipelines. 2.6. Model Evaluation Model performance was assessed using multiple evaluation metrics computed on each data partition, with test partition evaluation conducted strictly once after all preprocessing parameters, hyperparameter configurations, and decision thresholds were fully finalized. While standard machine learning studies routinely report receiver operating characteristic and precision–recall curves to characterize global performance across all possible operating thresholds, clinical deployment requires evaluating classifiers at specific, actionable operating thresholds. To support full class imbalance evaluation without relying on threshold-agnostic curves, confusion matrices were constructed to report exact raw true positive (TP), true negative (TN), false positive (FP), and false negative (FN) counts for all models across partitions, enabling readers to directly inspect operating point metrics. The primary metrics derived from these elements included accuracy, precision, recall (sensitivity), specificity, F1 score, and area under the receiver operating characteristic curve (AUC-ROC). The formal computation procedure for all binary classification metrics is specified in Algorithm A5. The F1 score, representing the harmonic mean of precision and recall, was computed as(9)F1=2·Precision·RecallPrecision+Recall=2·TP2·TP+FP+FN Beyond these foundational measures, a diverse suite of composite metrics and methodological extensions, each designed with specific clinical meaning and purpose, were utilized to provide a multidimensional assessment of model utility in high-stakes healthcare contexts [24,25]. 2.7. Learning Curve Analysis Learning curves were constructed to assess model behavior as a function of training set size and to diagnose potential overfitting or underfitting [26,27,28]. For each model, nested stratified subsets of the training data ranging from 10 to 100 percent were created at 10 percentage point increments. The nested stratified subset procedure independently permuted indices within each class, then selected the first ⌊f·nc⌋ observations from each permuted class c, where f denotes the target fraction, and nc denotes the class size. This approach ensures class proportions remain consistent across all training subset sizes. At each subset size, the model was trained multiple times using different random seeds (10 repetitions for CSBT models, 20 repetitions for RUSBoost), and both training and validation performance metrics were recorded. The mean and standard deviation of accuracy across repetitions were computed for each training fraction, enabling assessment of learning stability and generalization capacity. Models exhibiting non-convergent or non-plateauing learning curves across incremental training fractions (indicating learning instability or unreliability for deployment) were pre-specified for disqualification from final model selection, regardless of standalone test set performance. The complete stratified learning curve generation procedure is formalized in Algorithm A6. Table 3 summarizes the learning curve analysis parameters. 2.8. Feature Importance Predictor importance was quantified using the permutation-based importance measure inherent to tree ensemble methods. For each predictor, the importance value reflects the total reduction in the splitting criterion (weighted by the probability of reaching each node) summed across all trees in the ensemble. Features with higher importance values contribute more substantially to the classification decisions [29,30]. The top-20 predictors ranked by importance were identified and visualized for each model. 2.9. Clinically Oriented Evaluation Protocol Traditional machine learning evaluation metrics, while mathematically rigorous, often fail to capture the clinical consequences that determine a model’s value for patient care. To address this limitation, we developed a clinically oriented evaluation protocol comprising four composite metrics and five methodological extensions. This framework translates clinical priorities into quantitative assessment criteria through adjustable weighting parameters, enabling evaluation tailored to specific medical contexts where the consequences of different errors vary substantially. 2.9.1. Core Composite Metrics The foundational framework comprises four composite metrics, each addressing a different aspect of clinical performance through weighted combination of traditional metrics. The selection of these four metrics for opioid misuse screening is grounded in the multidimensional nature of clinical decision making in this domain. Screening applications require simultaneous consideration of (i) overall discriminative capacity to rank individuals by risk, (ii) threshold-specific detection rates that determine how many cases are identified versus missed, (iii) the interpretability of positive and negative predictions for downstream clinical action, and (iv) the asymmetric consequences of different error types. No single conventional metric addresses all four considerations; accordingly, we construct a complementary suite wherein CDPS captures discriminative performance with adjustable sensitivity-specificity emphasis, CPUS characterizes predictive value from the perspective of acting on individual predictions, WEAS quantifies class-specific accuracy with endpoint prioritization, and CEPM directly encodes asymmetric misclassification costs. Together, these metrics span the evaluation dimensions relevant to clinical screening while permitting context-specific parameterization through their respective weighting schemes. Clinical Discriminative Performance Score (CDPS) The CDPS combines area under the receiver operating characteristic curve (AUC), sensitivity, and specificity with adjustable weights reflecting clinical priorities:CDPS=α1·AUC+α2·Sensitivity+α3·Specificity where α1+α2+α3=1, and each αi≥0. The theoretical justification for combining these three metrics derives from their complementary roles in characterizing classifier performance. AUC is a threshold-independent measure that quantifies the probability that a randomly selected positive instance will be ranked higher than a randomly selected negative instance, thereby capturing the classifier’s overall discriminative capacity across the full range of possible operating points. However, AUC alone provides no guidance regarding performance at any specific decision threshold and can be misleading when class distributions differ between development and deployment contexts. Sensitivity and specificity, by contrast, are threshold-dependent metrics that characterize performance at a particular operating point, directly quantifying the conditional probabilities of correct classification given true class membership. Their inclusion ensures that the composite metric reflects clinically actionable performance rather than abstract rank-ordering ability. The weighted linear combination follows from multi-attribute utility theory, which establishes that when component attributes are mutually preferentially independent, a decision maker’s preferences can be represented as an additive value function over the individual attributes. Under this framework, the weights αi encode the relative importance of each attribute, transforming the multidimensional performance space into a scalar index that preserves the decision maker’s preference ordering. Specifically, the AUC component captures overall discriminative ability across all classification thresholds, while sensitivity and specificity components allow differential emphasis on positive or negative class detection. Formally, for non-trivial classifiers, CDPS∈[0.5α1,1.0]. The composite combines threshold-invariant discriminative ranking (AUC, weighted by α1) with operating-point true positive rate (Sensitivity=TP/(TP+FN), weighted by α2) and true negative rate (Specificity=TN/(TN+FP), weighted by α3). This formulation ensures that overall discrimination and operating-point clinical priorities are evaluated simultaneously. The weight parameters α1, α2, and α3 encode clinical priorities as importance coefficients. The constraint ∑i=13αi=1 ensures that CDPS remains on a normalized scale, while the non-negativity constraint αi≥0 ensures that each component contributes positively to overall performance. The ratio α2/α3 directly expresses the relative clinical importance of detecting positive cases versus avoiding false alarms; when α2/α3>1, sensitivity is prioritized over specificity. For screening applications where identifying positive cases takes precedence, sensitivity receives elevated weight (e.g., α1=0.20, α2=0.60, α3=0.20). For confirmatory applications where ruling in disease requires high confidence, specificity receives elevated weight (e.g., α1=0.20, α2=0.20, α3=0.60). Table 4 presents three weight configurations evaluated in this study. Clinical Predictive Utility Score (CPUS) The CPUS combines positive predictive value (PPV), negative predictive value (NPV), and F1 score to assess what predictions mean for individual patients:CPUS=ω1·PPV+ω2·NPV+ω3·F1 where ω1+ω2+ω3=1, and each ωi≥0. The appropriateness of CPUS for opioid misuse screening derives from its focus on the clinician’s epistemic state after observing a prediction. In screening workflows, the actionable question is not “given that a patient has opioid misuse history, will the model detect it?” (sensitivity), but rather “given that the model flags this patient, how confident should I be that intervention is warranted?” (PPV). Similarly, when the model issues a negative prediction, the clinician must judge whether follow-up assessment can be safely deferred (NPV). The F1 component provides a harmonic summary that penalizes extreme imbalances between precision and recall, ensuring that models achieving high PPV through overly conservative prediction thresholds are appropriately penalized. Unlike sensitivity and specificity, which characterize test performance conditional on true disease status, predictive values characterize the probability of disease conditional on test result. PPV answers the question: given a positive prediction, what is the probability that the patient truly has the condition? NPV answers the complementary question for negative predictions. Formally, CPUS∈[0,1.0]. The composite combines positive predictive value (PPV=TP/(TP+FP), weighted by ω1), negative predictive value (NPV=TN/(TN+FN), weighted by ω2), and the F1 score (weighted by ω3). Because PPV depends directly on base-rate prevalence π via Bayes’ theorem (frequently suppressing precision in low-prevalence screening), this formulation measures post-test probability and predictive reliability from the clinician’s perspective while penalizing extreme precision-recall imbalances. The weight parameters ω1, ω2, and ω3 encode the relative importance of positive prediction accuracy, negative prediction accuracy, and balanced performance, respectively. The ratio ω1/ω2 expresses the relative clinical consequence of acting on positive versus negative predictions. In screening contexts where positive predictions trigger resource-intensive workup, ω1 is elevated to ensure that models generating many false alarms are penalized appropriately. For applications where positive predictions trigger resource-intensive follow-up, PPV receives elevated weight (e.g., ω1=0.60, ω2=0.20, ω3=0.20). For applications where negative predictions are used to rule out disease, NPV receives elevated weight (e.g., ω1=0.20, ω2=0.60, ω3=0.20). Table 5 presents three weight configurations evaluated in this study. Weighted Endpoint Accuracy Score (WEAS) For ordinal classification problems where classes represent severity levels, the WEAS emphasizes accurate classification of clinically critical endpoint classes while still accounting for intermediate class performance:WEAS=γend·Accendpoints+γmid·Accintermediate where γend+γmid=1, Accendpoints represents the average accuracy for the endpoint classes (e.g., healthy and severe disease), and Accintermediate represents the average accuracy for intermediate classes. For binary classification problems such as the opioid misuse prediction task, both classes are treated as endpoints, and WEAS reduces to a weighted average of class-specific accuracies:WEASbinary=γpos·Sensitivity+γneg·SpecificityThis formulation is equivalent to balanced accuracy when γpos=γneg=0.5, but it allows differential weighting when one endpoint carries greater clinical significance. For opioid misuse screening, WEAS is particularly appropriate because it directly addresses the class imbalance problem inherent in this domain. Standard accuracy would allow a model to achieve nearly 90% performance by trivially predicting all-negative, thereby providing no clinical value. WEAS, by separately weighting accuracy on the positive class (sensitivity) and the negative class (specificity) and then combining them, ensures that minority class performance contributes proportionally to the composite score regardless of prevalence. This property aligns with the screening objective of identifying individuals with opioid misuse history, who constitute only 11.7% of the population in the present study. The WEAS admits the following formal interpretation. For the binary formulation, let Sensitivity,Specificity∈[0,1] and weights γpos,γneg≥0 with γpos+γneg=1. Then, WEASbinary∈[0,1], achieving its maximum of 1 when both sensitivity and specificity equal 1. The metric can be rewritten as:WEASbinary=γpos·TPTP+FN+γneg·TNTN+FPThis formulation reveals that WEAS computes a weighted average of class-conditional accuracies, where each class contributes according to its designated importance rather than its prevalence. The weight ratio γpos/γneg directly encodes the relative importance of correctly classifying positive versus negative cases. When γpos/γneg>1, the metric prioritizes minority class detection; when γpos/γneg<1, it prioritizes majority class accuracy. For the general ordinal formulation, let Accendpoints denote the average accuracy for endpoint classes (e.g., healthy and severe disease) and Accintermediate denote the average accuracy for intermediate classes. The weighting scheme γend>γmid reflects the clinical judgment that misclassification at the extremes of the severity spectrum carries greater consequence than misclassification among intermediate categories. Clinical Endpoint Performance Metric (CEPM) The CEPM extends WEAS by penalizing misclassifications through a trade-off parameter λ∈[0,1] that balances class-balanced accuracy against overall error rate. For binary classification, the CEPM formula evaluated across the empirical analysis is specified as:(10)CEPM=λ·WEAS−(1−λ)·Penalty where WEAS=(Sensitivity+Specificity)/2 represents the balanced accuracy, Penalty=(FP+FN)/n denotes the overall misclassification rate, and n is the total sample size. When λ=0.5, CEPM assigns equal weight to balanced accuracy and error minimization; higher values of λ prioritize balanced class performance, whereas lower values prioritize raw error rate reduction. The appropriateness of CEPM for opioid misuse screening stems from its capacity to trade off minority class detection against overall error minimization. In opioid misuse prediction, class imbalance means raw error rates are dominated by majority-class true negatives. By controlling the weight λ assigned to balanced accuracy (WEAS) relative to overall misclassification rate (Penalty), CEPM allows model evaluation to shift from raw error minimization to balanced per-class accuracy, enabling direct comparison of classifiers under varying clinical priorities. The CEPM admits the following formal interpretation. By expanding WEAS=(Sensitivity+Specificity)/2 and Penalty=(FP+FN)/n, the metric explicitly trades off balanced per-class accuracy against overall error rate. The metric achieves its theoretical maximum when WEAS=1 and Penalty=0, yielding CEPM=λ. Under this parameterization, the balancing parameter λ directly operationalizes the clinical choice between prioritizing minority class detection versus minimizing total error count. The parameter λ∈[0,1] defines the relative emphasis placed on class-balanced performance versus total error minimization: when λ=0.5, equal weight is assigned to both dimensions; when λ>0.5, higher priority is assigned to class-balanced accuracy (WEAS priority), which is appropriate for screening contexts where detecting minority positive cases is essential; when λ<0.5, higher priority is assigned to overall error reduction (Penalty priority). Table 6 presents the CEPM weight configurations evaluated in this study. 2.9.2. Expert Weight Elicitation To address the subjectivity inherent in weight selection, a systematic weight elicitation procedure was implemented using swing weight methodology. This approach transforms subjective clinical judgment into documented, defensible weight selections through structured comparison of metric importance. Before presenting the elicitation methodology, we establish the theoretical and clinical foundations that constrain the plausible weight space. The weights employed in this study are grounded in decision-analytic principles and the clinical characteristics of opioid misuse as a screening target. From a decision-analytic perspective, multi-attribute utility theory establishes that weights should reflect the relative value of improving each attribute from its worst to best plausible level. For screening applications where the consequences of missed cases exceed the consequences of false alarms, sensitivity weights should exceed specificity weights. This theoretical constraint bounds the ratio α2/α3 to values greater than unity for screening contexts. The clinical characteristics of opioid misuse strongly favor sensitivity prioritization. Opioid use disorder is a progressive condition: individuals who misuse prescription opioids face elevated risk of tolerance, dependence, and eventual addiction, with continued use leading to both short-term physiological effects and long-term damage to vital organs. The most severe consequence of unidentified opioid misuse is overdose, which has become a leading cause of preventable death. These outcomes are largely irreversible once they occur, whereas the consequences of a false positive in screening are comparatively modest, typically consisting of additional clinical assessment that ultimately confirms no intervention is needed. This asymmetry in consequence severity provides strong clinical justification for weighting sensitivity more heavily than specificity. The economic burden of opioid misuse further supports sensitivity prioritization. Individuals with opioid use disorder experience substantially elevated rates of healthcare utilization, lost workplace productivity, and social service involvement. Early identification enables targeted intervention before these costs accumulate, whereas missed cases allow the condition to progress. The cost differential between untreated opioid use disorder and unnecessary screening follow-up typically favors aggressive case identification by a substantial margin. These considerations converge on a plausible weight region for opioid misuse screening: α1∈[0.20,0.35] to ensure overall discriminative quality is rewarded, α2∈[0.40,0.55] to prioritize case identification consistent with the severity of missed-case consequences, and α3∈[0.15,0.30] to maintain appropriate but lower emphasis on false alarm minimization. The hypothetical expert elicitation presented below demonstrates the swing weight methodology within this clinically grounded space; the resulting weights should be interpreted as illustrative parameterizations consistent with the clinical profile of opioid misuse rather than as arbitrary selections. The swing weight procedure involves three steps:Anchor identification: The metric considered most important for the clinical context is assigned a reference weight of 100.Relative weighting: Each remaining metric is assigned a weight between 0 and 100 reflecting its importance relative to the anchor based on the question: “If only one metric could improve from its worst to best plausible value, how important would improvement in this metric be compared to the anchor?”Normalization: Raw weights are normalized to sum to 1.0. Anchor identification: The metric considered most important for the clinical context is assigned a reference weight of 100. Anchor identification: The metric considered most important for the clinical context is assigned a reference weight of 100. Relative weighting: Each remaining metric is assigned a weight between 0 and 100 reflecting its importance relative to the anchor based on the question: “If only one metric could improve from its worst to best plausible value, how important would improvement in this metric be compared to the anchor?” Relative weighting: Each remaining metric is assigned a weight between 0 and 100 reflecting its importance relative to the anchor based on the question: “If only one metric could improve from its worst to best plausible value, how important would improvement in this metric be compared to the anchor?” Normalization: Raw weights are normalized to sum to 1.0. Normalization: Raw weights are normalized to sum to 1.0. To illustrate application of the swing weight methodology, we constructed five hypothetical expert profiles representing the range of stakeholder perspectives relevant to opioid misuse screening. These profiles are not intended to substitute for actual expert elicitation but rather to demonstrate how the methodology operates and to show that the resulting weights fall within the clinically justified range established above. The addiction medicine specialist profile prioritizes case identification because early intervention can prevent progression from misuse to opioid use disorder and reduce overdose risk. The primary care physician profile balances sensitivity and specificity given workflow constraints in general practice settings where follow-up burden must be managed. The epidemiologist profile prioritizes AUC as a threshold-independent summary of overall model quality, assigning equal importance to sensitivity and specificity from a population health perspective. The quality improvement officer profile prioritizes specificity to minimize unnecessary follow-up burden and associated healthcare costs. The patient advocate profile strongly prioritizes sensitivity to ensure that no individual with opioid misuse history is overlooked, reflecting the perspective that missed cases represent failed opportunities for potentially life-saving intervention. The diversity of these profiles is intentional: it captures the range of legitimate clinical priorities and enables assessment of weight sensitivity across this range. Importantly, even the profile most favorable to specificity (quality improvement officer) assigns weights within the theoretically justified bounds, ensuring that all profiles represent clinically defensible positions. Individual weight vectors were aggregated using the linear opinion pool:αiaggregated=1K∑k=1Kαi(k) where K=5 is the number of experts, and αi(k) is expert k’s normalized weight for component i. Inter-expert agreement was quantified using the coefficient of variation (CV) across expert weights for each component. Components with CV<0.20 were considered to have high agreement; components with CV>0.40 were flagged for sensitivity analysis. The sensitivity analysis framework ensures that conclusions are robust across the full range of clinically justified weights, thereby reducing dependence on any single weight parameterization. 2.9.3. Uncertainty Quantification Bootstrap resampling was employed to quantify uncertainty in composite metric estimates arising from finite sample size and weight imprecision. For each of B=2000 bootstrap iterations:A bootstrap sample of size n was drawn with replacement from the test set.Confusion matrix elements were recomputed on the bootstrap sample.Component metrics (AUC, sensitivity, specificity, PPV, NPV, F1) were calculated.Composite scores (CDPS, CPUS, WEAS, CEPM) were computed using the point estimate weights. A bootstrap sample of size n was drawn with replacement from the test set. A bootstrap sample of size n was drawn with replacement from the test set. Confusion matrix elements were recomputed on the bootstrap sample. Confusion matrix elements were recomputed on the bootstrap sample. Component metrics (AUC, sensitivity, specificity, PPV, NPV, F1) were calculated. Component metrics (AUC, sensitivity, specificity, PPV, NPV, F1) were calculated. Composite scores (CDPS, CPUS, WEAS, CEPM) were computed using the point estimate weights. Composite scores (CDPS, CPUS, WEAS, CEPM) were computed using the point estimate weights. The 2.5th and 97.5th percentiles of the bootstrap distribution provided 95% confidence intervals for each composite metric. Additionally, weight uncertainty was incorporated by sampling weights from distributions centered on the expert-elicited values:αi(b)∼Beta(μi·κ,(1−μi)·κ) where μi is the elicited weight, and κ is a concentration parameter controlling weight precision. Following each draw, weights were renormalized to sum to 1.0. This joint resampling procedure captures both sampling variability and weight uncertainty in the reported confidence intervals. 2.9.4. Custom Penalty Matrices To evaluate model performance under clinically realistic cost assumptions, a custom penalty matrix framework was implemented. The penalty matrix C assigns differential costs to false negative and false positive errors:C=0cFPcFN0 where cFN is the cost of a false negative and cFP is the cost of a false positive. The expected cost for a classifier is:EC=cFP·FP+cFN·FN Five penalty scenarios were evaluated, with cost ratios ranging from symmetric (1:1) to highly asymmetric (10:1), as presented in Table 7. Crossover analysis identified the cost ratio at which optimal model selection changes. For models A and B, the crossover point is:cFN*=FPB−FPAFNA−FNBThis threshold has direct clinical interpretation: when the cost of missing a positive case exceeds cFN* times the cost of a false alarm, model B becomes cost-optimal. 2.9.5. Utility-Weighted Evaluation A utility-weighted extension incorporates explicit outcome valuations into the evaluation framework. Let UTP, UFP, UTN, and UFN denote the utilities of the four confusion matrix outcomes. The expected utility of classifier deployment is:EU=P(TP)·UTP+P(FP)·UFP+P(TN)·UTN+P(FN)·UFN where the outcome probabilities depend on model performance and disease prevalence:P(TP)=π×SensitivityP(FN)=π×(1−Sensitivity)P(TN)=(1−π)×SpecificityP(FP)=(1−π)×(1−Specificity) where π is the disease prevalence. This formulation enables model comparison on a scale directly reflecting clinical value, accounting for the base rate of disease in the target population. Three utility scenarios were evaluated: moderate asymmetry (UFN/UFP=2.5), high false negative aversion (UFN/UFP=10), and balanced utilities (UFN/UFP=1). Table 8 presents the utility parameterizations. 2.9.6. Net Benefit and Decision Curve Analysis Net benefit analysis relates model performance to treatment decision thresholds. The treatment threshold pt represents the disease probability at which a decision maker would be indifferent between intervention and no intervention, implicitly encoding the ratio of harms from unnecessary treatment to harms from missed disease. The net benefit at threshold pt is:NB(pt)=TPn−FPn·pt1−pt where n is the total sample size. The term pt/(1−pt) converts false positive counts to the scale of true positives based on their relative disvalue at the specified threshold. Decision curve analysis plots net benefit against treatment threshold across a clinically plausible range (0.05 to 0.50), comparing each classifier to default strategies of “treat all” (intervene for every patient) and “treat none” (intervene for no patient). A classifier adds clinical value when its net benefit exceeds both default strategies. The integrated net benefit (INB) over a threshold range quantifies aggregate clinical utility:INB=∫ptminptmaxNB(pt)dpt 2.9.7. Cost-Effectiveness Analysis Cost-effectiveness analysis extends utility-weighted evaluation by incorporating resource costs alongside health outcomes [31]. Total cost per patient screened was computed as:Ctotal=Cscreen+P(Pos)·Cfollowup+P(TP)·Cintervention+P(FN)·Cuntreated where P(Pos)=P(TP)+P(FP) is the probability of a positive screen. Health outcomes were measured in quality-adjusted life years (QALYs):Qtotal=P(TP)·QTP+P(FP)·QFP+P(TN)·QTN+P(FN)·QFN The incremental cost-effectiveness ratio (ICER) comparing strategy A to strategy B is:ICERAvsB=CA−CBQA−QBA strategy is cost-effective if its ICER falls below a willingness-to-pay threshold (conventionally $50,000 to $100,000 per QALY in the United States). Probabilistic sensitivity analysis was conducted using Monte Carlo simulation with 10,000 iterations. Cost parameters were sampled from gamma distributions with coefficient of variation 0.25; QALY impacts were sampled from beta distributions with coefficient of variation 0.30. Cost-effectiveness acceptability curves were generated showing the probability that each strategy is cost-effective as a function of willingness-to-pay threshold. 2.9.8. Model Ranking Concordance To assess agreement across evaluation frameworks, model rankings were compared across all metrics and scenarios. For each pair of evaluation approaches, concordance was quantified as:Concordance=1.0ifsamemodelrankedfirst0.5ifsamemodelrankedfirstorsecond0.0ifdifferentmodelsrankedfirst A concordance matrix was constructed to identify which evaluation frameworks yield consistent recommendations and which diverge. Divergence between frameworks is not necessarily problematic; rather, it reflects the distinct clinical questions each framework addresses. CDPS emphasizes discriminative ability independent of prevalence, while expected utility incorporates prevalence through probability weights. CPUS emphasizes predictive values relevant to individual patient interpretation, while net benefit emphasizes population-level treatment decisions. Understanding these distinctions enables appropriate framework selection for specific evaluation contexts. 2.10. Software Implementation All analyses were conducted using MATLAB (version R2026a, MathWorks, Natick, MA, USA) with the Statistics and Machine Learning Toolbox (version R2026a). The fitcensemble function was employed for training both CSBT and RUSBoost models, with the Method parameter set to ‘Bag’ for bagging and ‘RUSBoost’ for the boosted undersampling approach. Decision tree base learners were specified using the templateTree function with configurable maximum split parameters. Performance curves were generated using the perfcurve function, and predictor importance was extracted using the predictorImportance function. 3. Results The evaluation results are presented in a structured, thematic sequence designed to move from foundational classification performance to clinical decision-analytic utility. First, raw baseline performance and confusion matrices are reported across partitions, followed by internal convergence diagnostics via learning curves and feature importance rankings. Next, the evaluation transitions to the clinically oriented protocol, systematically analyzing discriminative performance score, predictive utility, weighted endpoint accuracy, and endpoint performance metrics. Finally, extended decision-analytic evaluations, including expert weight elicitation, uncertainty quantification, penalty matrix crossover analysis, decision curve net benefit, and cost-effectiveness modeling, are detailed to establish deployment readiness. 3.1. Classification Performance Across Models Figure 6 presents the confusion matrices for all three classification models evaluated on training, validation, and test partitions. The CSBT-Untuned model achieved an overall accuracy of 86.7% on the test set, correctly classifying 1592 true negatives and 87 true positives. The model produced 139 false negatives, corresponding to a sensitivity of 38.5% for individuals with documented opioid misuse history. The CSBT-Tuned model, employing an optimized decision threshold of 0.33, achieved 116 true positives and 110 false negatives on the test set. This represents a recall of 51.3%, a relative improvement over the untuned variant, while overall accuracy was 83.6%. The RUSBoost model achieved 85.1% test accuracy with 108 true positives and 118 false negatives, corresponding to a recall of 47.8%. Across all three models, validation set performance was consistent with test set results. Table 9 summarizes the performance metrics for each model. 3.2. Learning Curve Analysis Figure 7 presents the learning curves for all three classification models across incremental training set sizes. At 10% of training data, the CSBT-Untuned model achieved 92.05% training accuracy and 81.85% validation accuracy. As training set size increased to 100%, training accuracy declined to 88.80%, while validation accuracy reached 88.40%. The CSBT-Tuned model displayed similar dynamics, with training accuracy reaching 89.05% and validation accuracy reaching 88.65% at full data utilization. For both CSBT variants, validation accuracy remained within 0.5 percentage points of its final value for all training fractions exceeding 40%. The RUSBoost model exhibited monotonic increases in both training and validation accuracy with increasing sample size. Accuracy rose from 79.70% (training) and 75.60% (validation) at 10% of data to 86.00% and 86.10% at 100% utilization, respectively. The difference between training and validation accuracy remained below 1.5 percentage points across all evaluated training fractions, with validation accuracy slightly exceeding training accuracy at the final 100% data point. 3.3. Feature Importance Analysis Feature importance values were extracted from the CSBT-Tuned model, which was identified as the preferred classifier based on its superior minority class detection (the only model achieving TP > FN on the test set) and stable learning curve convergence characteristics. Figure 8 presents the top-eight predictors ranked by permutation-based importance. Lifetime alcohol use (alcever) emerged as the most influential predictor (relative importance = 1.00), followed by lifetime heroin use (herever, 0.75). Behavioral substance use history variables demonstrated higher predictive contribution than demographic or clinical factors. Age category (catage, 0.70) and marital status (irmarit, 0.66) demonstrated comparable importance. Major depressive episode in the past year (amdeyr) was the highest-ranked mental health indicator with a relative importance of 0.43. Receipt of substance use treatment in the past year (txyrrecvd2) and race/ethnicity (newrace2) achieved relative importance values of 0.30 and 0.20, respectively. The demographic variable for sex (irsex, 0.07) demonstrated the lowest relative importance among the top eight predictors. 3.4. Clinical Discriminative Performance Analysis To evaluate model performance within a clinically oriented framework, the Clinical Discriminative Performance Score (CDPS) was computed for each model under three distinct clinical priority scenarios. The CDPS integrates AUC, sensitivity, and specificity into a single weighted index according toCDPS=α1·AUC+α2·Sensitivity+α3·Specificity where the weighting coefficients satisfy α1+α2+α3=1. To maintain comparability across performance dimensions at the selected operating point, the balanced accuracy (Sensitivity+Specificity)/2 was employed as the AUC component in CDPS calculations, yielding values of 0.658 for CSBT-Untuned, 0.696 for CSBT-Tuned, and 0.690 for RUSBoost. This threshold-specific measure was chosen because it reflects discriminative capacity at the clinically selected decision boundary. For reference, full threshold-agnostic ROC AUC values computed from continuous prediction probabilities were 0.724 for CSBT-Untuned, 0.741 for CSBT-Tuned, and 0.738 for RUSBoost; these values confirm that model rankings are preserved regardless of the discrimination metric employed. Table 10 presents CDPS values under three weighting scenarios: balanced evaluation (α1=0.34, α2=0.33, α3=0.33), sensitivity priority (α1=0.20, α2=0.60, α3=0.20), and specificity priority (α1=0.20, α2=0.20, α3=0.60). The sensitivity priority scenario reflects clinical contexts where identifying individuals at risk for opioid misuse is paramount, whereas the specificity priority scenario reflects contexts where minimizing false alarms takes precedence. Under balanced evaluation, CSBT-Tuned achieved the highest CDPS (0.696), followed by RUSBoost (0.690) and CSBT-Untuned (0.658). This ordering was preserved under sensitivity priority weighting, where CSBT-Tuned attained a CDPS of 0.623 compared to 0.605 for RUSBoost and 0.549 for CSBT-Untuned. The relative improvement of CSBT-Tuned over CSBT-Untuned under sensitivity priority conditions was 13.5%. Under specificity priority weighting, the ranking shifted: RUSBoost achieved the highest CDPS (0.775), followed by CSBT-Tuned (0.769) and CSBT-Untuned (0.768). The differences among models under this scenario were substantially smaller than under sensitivity priority conditions, with all three models achieving CDPS values within 0.007 of each other. Figure 9 illustrates the CDPS values across all three scenarios. The crossover pattern is evident: CSBT-Tuned attains the highest point estimates under balanced and sensitivity priority conditions (though closely matched by RUSBoost), whereas the three models converge to near-equivalent performance under specificity priority conditions. This pattern reflects the distinct sensitivity and specificity profiles of the three models and demonstrates that optimal model selection depends on the clinical priorities encoded in the weighting scheme. 3.5. Clinical Predictive Utility Analysis To complement the discriminative performance analysis, the Clinical Predictive Utility Score (CPUS) was computed for each model under three clinical priority scenarios. The CPUS integrates Positive Predictive Value (PPV), Negative Predictive Value (NPV), and F1 score into a single weighted index according toCPUS=ω1·PPV+ω2·NPV+ω3·F1 where the weighting coefficients satisfy ω1+ω2+ω3=1. This metric shifts focus from discriminative ability to predictive utility, addressing the question of what a positive or negative prediction means for the individual patient. Table 11 and Figure 10 present the component metrics and composite CPUS values under three weighting scenarios: balanced evaluation (ω1=0.34, ω2=0.33, ω3=0.33), PPV priority (ω1=0.60, ω2=0.20, ω3=0.20), and NPV priority (ω1=0.20, ω2=0.60, ω3=0.20). The PPV priority scenario reflects clinical contexts where positive predictions trigger resource-intensive follow-up such that minimizing false alarms takes precedence. The NPV priority scenario reflects contexts where confidently ruling out opioid misuse history is essential. Examination of the component metrics revealed substantial variation in PPV across models, with CSBT-Untuned achieving the highest value (0.422), followed by RUSBoost (0.388) and CSBT-Tuned (0.359). NPV was uniformly high across all models, ranging from 0.920 to 0.932, reflecting the low prevalence of opioid misuse in the sample (approximately 11.6%). Figure 11 illustrates this decomposition, revealing that model differences in CPUS arise primarily from variation in PPV rather than NPV. Under balanced evaluation, CSBT-Untuned and RUSBoost achieved equivalent CPUS values (0.580), with CSBT-Tuned achieving a modestly lower score (0.569). Under PPV priority weighting, CSBT-Untuned achieved the highest CPUS (0.518), representing a 6.6% relative improvement over CSBT-Tuned (0.486). Under NPV priority weighting, all three models converged to similar performance, with RUSBoost achieving a marginal advantage (0.721 versus 0.717 for CSBT-Untuned and 0.715 for CSBT-Tuned). Figure 10 illustrates the CPUS values across all three scenarios. The pattern observed differs from that of the CDPS analysis: whereas CSBT-Tuned achieved superior CDPS under sensitivity priority conditions, it achieved inferior CPUS under PPV priority conditions. This reversal reflects the fundamental trade-off between sensitivity and positive predictive value. The threshold reduction that increased CSBT-Tuned sensitivity from 38.5% to 51.3% simultaneously increased false positives from 119 to 207, reducing PPV from 42.2% to 35.9%. 3.6. Weighted Endpoint Accuracy Analysis The Weighted Endpoint Accuracy Score (WEAS) framework was applied to evaluate model performance with equal weighting of both outcome classes. For binary classification problems, the WEAS formula simplifies to balanced accuracy, computed as the arithmetic mean of per-class accuracies:WEAS=Acc0+Acc12 where Acc0 represents the accuracy on the negative class (specificity), and Acc1 represents the accuracy on the positive class (sensitivity). This formulation weights both classes equally regardless of their prevalence in the dataset, providing a complement to raw accuracy that can be dominated by majority class performance in imbalanced settings. Table 12 presents the per-class accuracies, raw accuracy, and WEAS for all three models. Examination of per-class accuracies revealed substantial disparities between majority and minority class performance across all models. CSBT-Untuned achieved the highest majority class accuracy (Acc0 = 0.931) but the lowest minority class accuracy (Acc1 = 0.385), yielding a disparity of 0.546. CSBT-Tuned achieved a more balanced profile (Acc0 = 0.879, Acc1 = 0.513), with a reduced disparity of 0.366. RUSBoost occupied an intermediate position (Acc0 = 0.901, Acc1 = 0.478), with a disparity of 0.423. The ranking of models differed substantially between raw accuracy and WEAS. Raw accuracy ranked CSBT-Untuned highest (0.867), followed by RUSBoost (0.851) and CSBT-Tuned (0.836). WEAS reversed this ordering, ranking CSBT-Tuned highest (0.696), followed by RUSBoost (0.690) and CSBT-Untuned (0.658). Figure 12 illustrates this reversal, demonstrating that the model with the lowest raw accuracy achieved the highest balanced accuracy. The gap between raw accuracy and WEAS provides a measure of how much a model’s aggregate performance derives from majority class dominance. This gap was largest for CSBT-Untuned (0.209), indicating that 24.1% of its raw accuracy score reflected the imbalanced class distribution rather than balanced classification ability. The gap was smallest for CSBT-Tuned (0.140), consistent with its more balanced per-class performance profile. Figure 13 decomposes the per-class accuracies, revealing the source of these disparities. The 5.8% relative improvement in WEAS achieved by CSBT-Tuned over CSBT-Untuned (0.696 versus 0.658) quantifies the benefit of threshold optimization when both classes are weighted equally. This improvement emerges from the 33.2% relative increase in minority class accuracy (0.513 versus 0.385) that threshold tuning provides, which more than compensates for the 5.6% relative decrease in majority class accuracy (0.879 versus 0.931). 3.7. Clinical Endpoint Performance Metric Analysis The Clinical Endpoint Performance Metric (CEPM) was computed to evaluate the trade-off between balanced accuracy and misclassification severity. The CEPM combines the WEAS with a distance-based penalty through a balancing parameter:CEPM=λ·WEAS−(1−λ)·Penalty where λ∈[0,1] controls the relative emphasis on balanced accuracy versus error minimization. For binary classification, the distance penalty simplifies to the misclassification rate, as all errors involve adjacent classes with unit distance. Table 13 presents the component metrics and CEPM values at four levels of λ. CSBT-Untuned achieved the lowest penalty (0.133), reflecting its high raw accuracy (86.7%), whereas CSBT-Tuned incurred the highest penalty (0.164), reflecting its lower raw accuracy (83.6%) resulting from threshold optimization. The WEAS values showed the opposite pattern, with CSBT-Tuned achieving the highest balanced accuracy (0.696) and CSBT-Untuned the lowest (0.658). The CEPM analysis revealed a crossover pattern in model rankings as λ varied. At λ=0.3 (penalty priority), CSBT-Untuned achieved the highest CEPM (0.104), followed by RUSBoost (0.103) and CSBT-Tuned (0.094). At λ=0.5 (balanced), RUSBoost achieved the highest CEPM (0.271), followed by CSBT-Tuned (0.266) and CSBT-Untuned (0.262). At λ=0.7 and λ=0.9 (WEAS priority), CSBT-Tuned achieved the highest CEPM, with RUSBoost in close second place. Figure 14 illustrates the CEPM as a continuous function of λ for all three models. The crossover point occurred at approximately λ=0.39: below this threshold, CSBT-Untuned achieved superior CEPM; above this threshold, CSBT-Tuned achieved superior CEPM. This crossover reflects the inverse relationship between the two component metrics. CSBT-Tuned achieved 5.8% higher WEAS but incurred 23.3% higher penalty relative to CSBT-Untuned. Figure 15 presents the trade-off between WEAS and penalty in a two-dimensional space. No model occupied the ideal upper-left region (high WEAS, low penalty), indicating that the models represented different positions along a Pareto frontier. The choice of optimal model depends on the slope of the indifference curves, which is determined by the balancing parameter λ. Figure 16 summarizes the CEPM rankings across four discrete values of λ, illustrating the shift in optimal model selection as clinical priorities change. 3.8. Integrated Clinical Metric Analysis To provide a detailed assessment, the four composite evaluation metrics were synthesized across all three models. Table 14 details these outcomes across representative clinical priority scenarios, which serve as the foundation for the final recommendations in Section 4.5. Comparative performance profiles are visualized in Figure 17, Figure 18 and Figure 19. 3.9. Systematic Weight Elicitation from Clinical Experts To demonstrate the establishment of weighting coefficients for the CDPS, a structured swing weight elicitation was simulated using a hypothetical panel of five expert profiles representing diverse stakeholder perspectives. The elicitation process followed the four-step swing weight methodology detailed in Section 2.9.2. As established in that section, the weights derived from this process are constrained by decision-analytic principles and the clinical characteristics of opioid misuse to lie within justified bounds. The primary contribution of this section is methodological: it demonstrates how structured elicitation transforms qualitative clinical priorities into quantitative weights suitable for composite metric calculation, providing a template that practitioners can adapt using actual expert panels in their specific deployment contexts. 3.9.1. Establishing Reference Bounds Based on performance ranges for opioid misuse prediction models and the observed performance of the current models, the following plausible bounds were established for the hypothetical screening context, summarized in Table 15. The asymmetric ranges reflect the clinical context of opioid misuse screening. The wider range for sensitivity (0.50) compared to specificity (0.28) indicates that sensitivity is expected to vary more substantially across models in this application, consistent with the observed challenge of detecting the minority positive class. 3.9.2. Individual Expert Elicitation Results Each hypothetical expert independently completed the swing weight procedure following the methodology in Section 2.9.2, with the resulting raw importance scores and normalized weights presented in Table 16. 3.9.3. Aggregated Weight Sets Three aggregation strategies were applied to derive weight sets for model evaluation. Each strategy is grounded in established principles from decision theory and multi-criteria aggregation methodology, with distinct assumptions about how expert influence should be distributed. Simple averaging assigns equal influence to all experts (wk=1/K for all k), computing the arithmetic mean across all five experts and yielding α1=0.28, α2=0.44, α3=0.29. This approach is statistically defensible under conditions of exchangeability, wherein no prior information distinguishes experts in terms of reliability, expertise, or relevance to the decision context [32]. Equal weighting avoids the introduction of arbitrary influence differentials and performs well in aggregation contexts where true relative expertise is unknown or difficult to quantify. From a reliability perspective, aggregating across multiple independent judgments reduces the influence of idiosyncratic error in any single expert’s response, analogous to the improvement in measurement reliability achieved through multi-item scales. The resulting weight set treats all expert perspectives as equally valid and produces a balanced representation of diverse priorities. Clinical weighting assigns differential influence based on relevance to the deployment context, giving greater weight to the three clinically trained experts (Experts 1, 2, and 5) relative to the administrative and methodological experts (Experts 3 and 4), with influence weights of 0.25, 0.25, 0.10, 0.10, and 0.30 respectively. The resulting aggregated weights were α1=0.25, α2=0.52, α3=0.23, reflecting stronger emphasis on sensitivity consistent with a screening application. This approach is defensible when the decision context privileges certain forms of expertise: for clinical screening applications, the priorities of practitioners who directly interact with patients and understand the consequences of missed cases carry greater normative weight than the priorities of those whose primary concerns are methodological or administrative. The influence weights represent a transparent meta-level judgment about whose priorities should predominate, which can itself be subjected to sensitivity analysis. Clinical weighting does not discard minority perspectives but rather tempers their influence in proportion to their relevance, preserving information about the range of stakeholder priorities while centering the aggregated weights on the clinical mission. Range-based evaluation preserved the full distribution of expert weights to enable sensitivity analysis [6]. The ranges were α1∈[0.21,0.42], α2∈[0.29,0.61], and α3∈[0.18,0.49]. This approach acknowledges that forcing consensus through aggregation may obscure meaningful disagreement and instead characterizes the space of defensible weight combinations to assess whether model selection conclusions are robust across this range. All aggregation methods employed satisfy the formal properties required of aggregation operators: boundary conditions (aggregated weights remain in [0,1]), averaging (aggregated weights fall within the convex hull of individual weights), commutativity (results are invariant to expert ordering), and monotonicity (increasing an individual weight increases the aggregated weight). The choice between simple averaging and clinical weighting represents a value judgment about stakeholder influence that should be documented and justified in each application context. Table 17 summarizes the aggregated weight sets alongside the predefined scenarios from the original evaluation. The elicited weights demonstrate that when clinical experts are asked to articulate priorities through a structured process, the resulting weights fall between the predefined balanced and sensitivity-priority scenarios. The simple average (α2=0.44) and clinical weighting (α2=0.52) both assign greater importance to sensitivity than the balanced scenario (α2=0.33) but less than the extreme sensitivity-priority scenario (α2=0.60). This finding validates the relevance of the predefined scenarios while suggesting that the sensitivity-priority scenario may slightly overweight sensitivity relative to expert consensus. The elicited weights fall within the clinically justified bounds established in Section 2.9.2. The simple average weights (α1=0.28, α2=0.44, α3=0.29) satisfy the constraints α1∈[0.20,0.35], α2∈[0.40,0.55], and α3∈[0.15,0.30], confirming that the hypothetical expert judgments are consistent with the clinical profile of opioid misuse as a screening target. The sensitivity priority reflected in these weights aligns with the progressive nature of opioid use disorder and the severity of consequences associated with missed cases, including continued substance misuse, escalation to addiction, and risk of fatal overdose. This convergence between top-down clinical reasoning and bottom-up elicitation results strengthens confidence that the weight parameterizations employed in this study represent defensible choices grounded in the characteristics of the target condition. 3.9.4. Scaling to Larger Expert Panels The linear opinion pool aggregation methodology generalizes directly to panels of arbitrary size. For a panel of K experts, the aggregated weight for component i is computed as:αiaggregated=∑k=1Kwk·αi(k),where∑k=1Kwk=1 with wk representing the influence weight assigned to expert k. Setting wk=1/K for all k yields simple averaging; alternative weighting schemes may assign greater influence to experts with specific domain expertise or clinical experience relevant to the deployment context. For larger panels, inter-expert agreement should be assessed using Kendall’s coefficient of concordance (W) in addition to the coefficient of variation, with W>0.70 typically indicating acceptable consensus. When substantial disagreement persists, subgroup analysis stratified by professional role (e.g., clinicians versus administrators) may reveal systematic priority differences warranting separate evaluation scenarios rather than forced aggregation. Practitioners implementing this framework should recruit expert panels reflecting the stakeholder diversity of their intended deployment context, with panel sizes of 7 to 15 members generally providing sufficient representativeness while remaining logistically feasible for structured elicitation procedures. 3.9.5. CDPS Under Elicited Weights Table 18 presents the CDPS values computed using the empirically elicited weight sets. Under both elicited weight sets, CSBT-Tuned achieved the highest CDPS, followed by RUSBoost and CSBT-Untuned. The model ranking was preserved across all elicitation-based aggregation strategies, indicating that the relative point estimate advantage of CSBT-Tuned is robust to the specific method of expert weight aggregation, while recognizing that CSBT-Tuned and RUSBoost display statistically overlapping performance. The relative advantage of CSBT-Tuned over CSBT-Untuned was 8.0% under simple averaging and 10.6% under clinical weighting, confirming the clinical value of threshold optimization. Figure 20 illustrates the CDPS values across predefined and elicited weight scenarios. 3.10. Uncertainty Quantification via Bootstrap Resampling To characterize the precision of performance estimates and enable statistically grounded model comparisons, bootstrap resampling was applied to the test set predictions. A total of B=2000 bootstrap samples were drawn with replacement from the 1937 test observations. For each bootstrap sample, all component metrics (sensitivity, specificity, AUC lower bound) and composite scores (CDPS under multiple weighting scenarios) were recomputed. Confidence intervals were constructed using the percentile method at the 95% level. 3.10.1. Component Metric Confidence Intervals Table 19 presents the point estimates and 95% bootstrap confidence intervals for the component metrics. The intervals for sensitivity are substantially wider than those for specificity, reflecting the smaller effective sample size for the positive class (n=226) compared to the negative class (n=1711). Sensitivity point estimates ranged from 0.385 [0.323, 0.451] for CSBT-Untuned to 0.513 [0.447, 0.580] for CSBT-Tuned, with standard errors (0.032 to 0.034) approximately five times larger than those for specificity (0.006 to 0.009). AUC lower-bound estimates for the threshold-optimized models (CSBT-Tuned and RUSBoost) ranged from 0.690 to 0.696, with complete results provided in the table. Crucially, the confidence intervals for sensitivity between CSBT-Tuned [0.447, 0.580] and RUSBoost [0.412, 0.544] exhibit substantial overlap, indicating that their apparent performance difference (0.513 vs. 0.478) may partly reflect sampling variability rather than a true population-level difference. Similarly, the AUC confidence intervals for CSBT-Tuned [0.661, 0.732] and RUSBoost [0.655, 0.724] overlap almost entirely, confirming comparable overall discriminative capacity between the two threshold-optimized approaches. In contrast, both threshold-optimized models demonstrate statistically distinct performance compared to the untuned baseline. 3.10.2. Composite Score Confidence Intervals Table 20 presents the bootstrap confidence intervals for CDPS across the five weighting scenarios. Under sensitivity priority weighting, the 95% CI for CSBT-Tuned was [0.579, 0.668] compared to [0.509, 0.590] for CSBT-Untuned. However, the sensitivity-priority CDPS interval for CSBT-Tuned [0.579, 0.668] overlaps substantially with that of RUSBoost [0.561, 0.650], showing that their composite score differences also carry sampling uncertainty. Under specificity priority weighting, the intervals for all three models converged to a range between 0.747 and 0.795. Figure 21 provides a forest plot visualization of these point estimates and their associated 95% confidence intervals across all evaluated clinical scenarios. 3.10.3. One-Dimensional Sensitivity Across Elicited Expert Weights To assess the robustness of model rankings to weight uncertainty, CDPS was computed across the full range of weights elicited from a hypothetical expert panel. The sensitivity weight α2 was varied from 0.29 (minimum expert value) to 0.61 (maximum expert value), with α1 and α3 adjusted proportionally to maintain the constraint α1+α2+α3=1 while preserving their relative ratio from the simple average (approximately 1:1 for AUC and specificity). Figure 22 displays the CDPS trajectories across this weight range. The sensitivity analysis demonstrates that CSBT-Tuned maintains the highest CDPS throughout the entire range of expert-elicited weights. At α2=0.29 (minimum expert weight for sensitivity), CSBT-Tuned achieved CDPS of 0.704 compared to 0.699 for RUSBoost and 0.686 for CSBT-Untuned. At α2=0.61 (maximum expert weight), CSBT-Tuned achieved CDPS of 0.665 compared to 0.648 for RUSBoost and 0.588 for CSBT-Untuned. The relative advantage of CSBT-Tuned over CSBT-Untuned increased from 2.6% at low sensitivity weights to 13.1% at high sensitivity weights, reflecting the superior sensitivity of the threshold-optimized model. Critically, the model ranking was preserved across all weight values within the expert-elicited range, indicating that the conclusion favoring CSBT-Tuned is robust to uncertainty in clinical weight specification. This robustness provides confidence that model selection based on CSBT-Tuned is not an artifact of particular weight choices, even as performance relative to RUSBoost remains within the bounds of sampling variability. 3.11. Multidimensional Parameter Space Sensitivity Analysis While the expert-elicited weight sets demonstrated ranking stability across discrete stakeholder profiles, a critical question is whether model rankings remain stable across the entire continuous space of clinically defensible parameterizations. To address this, a formal multidimensional sensitivity analysis was conducted, examining model rankings under systematic grid-based enumeration (17,885 weight combinations) and Monte Carlo simulation (10,000 iterations) across the CDPS weight parameters α1, α2, and α3. The analysis employed two complementary approaches: a grid-based enumeration of 17,885 weight combinations satisfying the constraint α1+α2+α3=1 within tolerance, and a Monte Carlo simulation of 10,000 randomly sampled weight vectors from the theoretically justified ranges (α1∈[0.20,0.35], α2∈[0.40,0.55], α3∈[0.15,0.30]). For each weight combination, CDPS was computed for all three models and rankings were determined. Figure 23 illustrates the CDPS distributions and their relationship to the sensitivity weight parameter α2. The results demonstrate highly robust model rankings across the plausible weight space. CSBT-Tuned achieved first place in all weight combinations under both grid-based and Monte Carlo analyses, with RUSBoost consistently ranking second and CSBT-Untuned ranking third. The CDPS distributions reveal clear separation between models: CSBT-Tuned achieved a mean CDPS of 0.649±0.012 across the Monte Carlo samples, compared to 0.635±0.014 for RUSBoost and 0.587±0.018 for CSBT-Untuned. The absence of distributional overlap between CSBT-Tuned and the other models indicates that the ranking conclusion is not sensitive to the specific weight values chosen within the theoretically justified bounds. Crossover analysis identified the weight thresholds at which model rankings would change. CSBT-Tuned is preferred over CSBT-Untuned whenever α2>0.008, a threshold far below the theoretical lower bound of 0.40. Similarly, CSBT-Tuned is preferred over RUSBoost whenever α2>0.084. Since the theoretically justified range for α2 spans [0.40,0.55], CSBT-Tuned dominates across the entire clinically defensible weight space. When the analysis was extended to the broader expert elicitation ranges (α1∈[0.21,0.42], α2∈[0.29,0.61], α3∈[0.18,0.49]), which encompass the full variability observed across hypothetical expert profiles, CSBT-Tuned retained first place in all 10,000 Monte Carlo samples. This finding confirms that the model selection conclusion is robust not only to variation within theoretically justified bounds but also to the broader range of weight values that might emerge from actual expert elicitation with diverse stakeholder panels. These results provide strong evidence that the selection of CSBT-Tuned as the preferred model for opioid misuse screening is not an artifact of the particular weight parameterization employed in the primary analysis. The model ranking conclusion is highly robust to variation in expert-elicited weights, supporting confidence in the practical applicability of this recommendation across clinical contexts that may differ in their precise prioritization of sensitivity, specificity, and overall discriminative ability. 3.12. Custom Penalty Matrix Analysis Using the custom penalty matrix framework and the five clinical scenarios (S1–S5) established in Section 2.9.4, the expected costs for each classifier were calculated across the test partition. Table 21 presents the results of these calculations, demonstrating how model rankings shift as the penalty for false negatives increases. Under symmetric penalties (S1), the expected cost reduces to the total number of misclassifications, and CSBT-Untuned achieved the lowest expected cost (258), followed by RUSBoost (288) and CSBT-Tuned (317). This ordering reflects the raw accuracy ranking, as symmetric costs weight all errors equally. As the false negative penalty increased, the ranking of models shifted. Under Scenario S2 (2:1 ratio), CSBT-Untuned retained the lowest expected cost (397), though the gap narrowed substantially. At Scenario S3 (3:1 ratio), a crossover occurred: RUSBoost achieved the lowest expected cost (524), with CSBT-Tuned (537) and CSBT-Untuned (536) achieving nearly equivalent costs. Under Scenarios S4 and S5, CSBT-Tuned consistently achieved the lowest expected cost, with values of 757 and 1307 respectively. Figure 24 presents the expected cost as a continuous function of the false negative penalty cFN, with cFP fixed at 1.0. The expected cost function for each model is linear in cFN:EC(cFN)=FP+cFN·FN The slopes of these lines are determined by the number of false negatives: CSBT-Untuned has the steepest slope (FN = 139), CSBT-Tuned has the shallowest slope (FN = 110), and RUSBoost is intermediate (FN = 118). The crossover points identify the cost ratios at which optimal model selection changes. The crossover analysis identified two critical thresholds. The first crossover, between CSBT-Untuned and RUSBoost, occurred at cFN=2.43, computed as:cFN*=FPRUS−FPUntFNUnt−FNRUS=170−119139−118=5121=2.43 The second crossover, between CSBT-Untuned and CSBT-Tuned, occurred at cFN=3.03:cFN*=FPTun−FPUntFNUnt−FNTun=207−119139−110=8829=3.03 These crossover values have direct clinical interpretation: when the cost of missing a case of opioid misuse exceeds approximately three times the cost of a false alarm, CSBT-Tuned becomes the cost-minimizing model choice. Figure 25 presents the expected cost values in a heatmap format, facilitating visual identification of optimal model selection across the penalty parameter space. To quantify the cost implications of model selection, the cost savings achievable by selecting the optimal model under each penalty scenario were computed. Table 22 presents these values, expressed both in absolute terms and as percentages relative to the worst-performing model. The cost savings analysis revealed that the penalty for suboptimal model selection was largest under extreme penalty scenarios. Under S5 (10:1 ratio), selecting CSBT-Untuned instead of CSBT-Tuned would incur 202 additional cost units, representing a 13.4% increase in expected cost. Conversely, under S1 (symmetric penalties), selecting CSBT-Tuned instead of CSBT-Untuned would incur 59 additional cost units, an 18.6% increase. The normalized cost per prediction provides a scale-independent measure of model efficiency. Under S3 (clinical screening ratio of 3:1), the normalized expected cost ranged from 270.5 per 1000 predictions (RUSBoost) to 277.2 per 1000 predictions (CSBT-Tuned). This narrow range indicates that at clinically plausible penalty ratios near the crossover threshold, model selection has modest impact on total expected cost. 3.13. Utility-Weighted and Decision-Analytic Evaluation 3.13.1. Expected Utility Calculations To evaluate model performance within a decision-analytic framework, expected utility (EU) was computed for each classifier under multiple utility parameterizations. The expected utility formulation integrates the probabilities of each confusion matrix outcome with their corresponding clinical valuations:EU=P(TP)·UTP+P(FP)·UFP+P(TN)·UTN+P(FN)·UFN The outcome probabilities were derived from model performance metrics and disease prevalence. At the observed test set prevalence of π=0.1167 (226 positive cases among 1937 observations), the probability calculations yielded the values presented in Table 23. Three utility scenarios were evaluated to reflect distinct clinical value structures, presented in Table 24. Scenario U1 (moderate asymmetry) represents a clinical context where false negatives carry approximately 2.5 times the disutility of false positives. Scenario U2 (high FN aversion) assigns false negatives ten times the disutility of false positives. Scenario U3 (balanced utilities) assigns equal disutility to both error types. Table 25 presents the expected utility calculations and the decomposition of component contributions for each model across the three clinical scenarios. Under Scenario U1 (moderate asymmetry), CSBT-Untuned achieved the highest total expected utility (0.7364), followed by RUSBoost (0.7236) and CSBT-Tuned (0.7089). This model ranking was preserved across Scenario U2 (high FN aversion) and Scenario U3 (balanced), as visualized in the comparative bar chart in Figure 26. To identify the utility parameterization at which model rankings would change, a crossover analysis was conducted; this determined that CSBT-Tuned becomes optimal only when the false negative disutility (UFN) exceeds −1.83. The mathematical basis for this crossover analysis relies on the expected utility difference between CSBT-Untuned and CSBT-Tuned, expressed as a function of the false negative disutility UFN:ΔEU=EUUnt−EUTun=ΔTN+ΔFP+ΔFN where the differences in component contributions depend on the utility values. Setting ΔEU=0 and solving for UFN yields the crossover point. 3.13.2. Net Benefit and Decision Curve Analysis Net benefit was computed for each model across a range of treatment thresholds pt from 0.05 to 0.50. The net benefit formula relates classifier performance to the exchange rate between true positive benefit and false positive harm:NB(pt)=TPn−FPn·pt1−pt The term pt/(1−pt) represents the odds at the treatment threshold, converting false positives to the scale of true positives based on their relative disvalue. At pt=0.10, a decision maker values avoiding one false positive as equivalent to 0.111 true positives; at pt=0.30, this exchange rate increases to 0.429. Table 26 presents net benefit calculations for all three models across selected treatment thresholds, along with the default strategies of treating all patients and treating none. At low treatment thresholds (pt≤0.15), CSBT-Tuned achieved the highest net benefit among classifiers. At pt=0.10, CSBT-Tuned’s net benefit of 0.0480 exceeded CSBT-Untuned (0.0381) by 26.0% and RUSBoost (0.0461) by 4.1%. This advantage reflects CSBT-Tuned’s superior sensitivity: when the threshold is low (indicating high tolerance for false positives), the benefit of additional true positive detections outweighs the cost of accompanying false positives. At intermediate thresholds (pt=0.20 to 0.30), RUSBoost achieved the highest net benefit, with values of 0.0339, 0.0266, and 0.0182 at thresholds 0.20, 0.25, and 0.30 respectively. At these thresholds, RUSBoost’s balanced sensitivity-specificity profile provided optimal performance. At high thresholds (pt≥0.40), all classifiers approached or fell below zero net benefit, indicating that the cost of false positives exceeded the benefit of true positives. The “Treat None” strategy achieved equal or superior net benefit beyond threshold 0.40 for CSBT-Tuned and beyond threshold 0.50 for the other models. Figure 27 presents the complete decision curve analysis across the clinically relevant threshold range. The decision curve analysis identified distinct ranges of clinical utility for each model. All three classifiers achieved positive net benefit across the threshold range 0.05 to approximately 0.45, demonstrating clinical value superior to either default strategy. The crossover between “Treat All” and the classifiers occurred at pt≈0.08, below which universal intervention achieves higher net benefit than classifier-guided intervention. The area between the classifier curve and the “Treat None” reference line quantifies the integrated net benefit across thresholds. Table 27 presents the integrated net benefit for each model over clinically relevant threshold ranges. RUSBoost achieved the highest integrated net benefit over the full screening range (0.10 to 0.30), with an INB of 0.00650 compared to 0.00621 for CSBT-Tuned and 0.00570 for CSBT-Untuned. Over the high-sensitivity range (0.10 to 0.20), CSBT-Tuned achieved the highest INB (0.00406), consistent with its superior performance at low thresholds. 3.13.3. Cost-Effectiveness Analysis A cost-effectiveness analysis was conducted to evaluate the economic implications of classifier-guided screening strategies. The analysis adopted a healthcare system perspective, incorporating direct costs of screening, follow-up assessment, and downstream healthcare utilization, along with health outcomes measured in quality-adjusted life years (QALYs). Table 28 presents the cost and effectiveness parameters used in the analysis. Parameter values were derived from published literature on substance use screening programs and opioid-related healthcare costs. The total cost per patient screened was computed as:Ctotal=Cscreen+P(Pos)×Cfollowup+P(TP)×Cintervention+P(FN)×Cuntreated where P(Pos)=P(TP)+P(FP) represents the probability of a positive screen. Total QALYs per patient were computed as:Qtotal=P(TP)×QTP+P(FP)×QFP+P(TN)×QTN+P(FN)×QFN Table 29 presents the cost-effectiveness calculations for each classifier and comparator strategies. All classifier-guided strategies achieved lower total costs and better health outcomes than no screening, indicating dominance of screening over non-intervention. Among classifiers, CSBT-Untuned achieved the lowest total cost ($754.88 per patient) due to its lower rate of positive screens requiring follow-up assessment. However, CSBT-Untuned also achieved the worst health outcomes (−0.00345 QALYs) due to its higher false negative rate. The incremental cost-effectiveness ratio (ICER) for moving from CSBT-Untuned to RUSBoost was $6463 per QALY gained. The ICER for moving from RUSBoost to CSBT-Tuned was $13,840 per QALY gained. Both values fall well below standard willingness-to-pay thresholds of $50,000 to $100,000 per QALY, indicating that the additional expenditure required for higher-sensitivity screening is cost-effective. Figure 28 presents the cost-effectiveness plane illustrating the positioning of each strategy. A probabilistic sensitivity analysis was conducted using Monte Carlo simulation with 10,000 iterations to characterize uncertainty in the cost-effectiveness estimates. Parameter values were sampled from distributions reflecting plausible uncertainty: costs were sampled from gamma distributions with coefficient of variation 0.25; QALY impacts were sampled from beta distributions with coefficient of variation 0.30. Figure 29 presents the cost-effectiveness acceptability curves showing the probability that each strategy is cost-effective across a range of willingness-to-pay thresholds. At willingness-to-pay thresholds below $15,000 per QALY, CSBT-Untuned had the highest probability of being cost-effective (58% at $10,000/QALY). At thresholds above $25,000 per QALY, CSBT-Tuned had the highest probability of being cost-effective, reaching 70% at $100,000/QALY. RUSBoost maintained an intermediate probability (20% to 25%) across most threshold values, reflecting its position between the two CSBT variants on both cost and effectiveness dimensions. 4. Discussion The results of this evaluation demonstrate that aggregate performance metrics, while mathematically rigorous, are often insufficient for assessing the practical utility of predictive models in high-stakes healthcare contexts. Detailed validation through multidimensional reporting is critical for ensuring that models are clinically appropriate and ready for deployment, as consolidated reporting guidelines emphasize the need for transparent, comprehensive evidence to judge clinical readiness [33,34]. Relying on final metrics is inadequate because no individual benchmark can fully reflect the clinical consequences of classification errors; therefore, reporting a suite of complementary metrics through composite performance profiles is necessary to align model evaluation with real-world clinical utility [35,36]. Furthermore, examining internal learning behavior (specifically assessing stability and convergence through learning curve analysis) provides vital diagnostics regarding a model’s learning dynamics that aggregate statistics simply cannot reveal [22,26,37]. This study serves as a template for such reporting, providing a framework that respects both the rigor of statistical analysis and the complexity of clinical decision making. 4.1. The Accuracy Paradox and Clinical Model Selection The results of this study provide a compelling illustration of the accuracy paradox in machine learning classification, i.e., a phenomenon wherein models achieving superior overall accuracy may paradoxically offer inferior practical utility for their intended application. As documented in the machine learning literature, accuracy fails to maintain reliability for imbalanced data modeling, potentially misleading model selection when minority class detection is the primary objective [38,39]. The CSBT-Untuned model’s 86.7% test accuracy initially appears superior to CSBT-Tuned’s 83.6%. However, this 3.1 percentage point accuracy advantage is achieved entirely through conservative classification that minimizes false positives at the expense of detecting true cases. With only 87 true positives against 139 false negatives, the model misses nearly two-thirds of individuals with opioid misuse history, which is a performance profile unsuitable for screening applications where identifying at-risk individuals is paramount. In contrast, the CSBT-Tuned model’s threshold optimization explicitly prioritizes minority class detection by maximizing F1 score rather than accuracy. The resulting model is unique among the three evaluated: it is the only classifier where true positives (116) exceed false negatives (110) on the held-out test set. This distinction is not merely statistical but carries direct clinical implications. For every 226 individuals with actual opioid misuse history in the test population, CSBT-Tuned correctly identifies 116 while missing 110, whereas CSBT-Untuned identifies only 87 while missing 139. The tuned model thus detects 29 additional cases per test-set-equivalent population, which is a 33% improvement in case detection that could translate to meaningful intervention opportunities in real-world deployment. The accuracy paradox observed here aligns with established findings that accuracy is not always the appropriate evaluation metric for classification problems involving rare events or imbalanced classes. When class distributions are skewed, as in opioid misuse prediction where positive cases constitute only 12% of observations, a model can achieve deceptively high accuracy by predominantly predicting the majority class [10,11,22]. The practical consequence is that “accurate” models may offer little value for their intended screening purpose. These findings support a broader principle in applied machine learning: model selection criteria must align with deployment objectives. For screening applications, targeting rare but consequential outcomes (such as identifying individuals at risk for opioid misuse) metrics emphasizing minority class detection (recall, F1 score, or application-specific cost functions) provide more meaningful guidance than overall accuracy. The 3.1% accuracy reduction incurred by threshold optimization represents an acceptable trade-off for the 33% improvement in positive case detection, particularly given the severe health consequences associated with undetected opioid misuse disorders. 4.2. Limitations and Future Work Several limitations warrant consideration. First, while CSBT-Tuned achieves the best minority class detection among the evaluated models, its 51.3% recall indicates that nearly half of positive cases remain undetected. This ceiling may reflect fundamental limitations in predicting complex behavioral outcomes from survey data, inherent noise in self-reported measures, or the absence of predictive features not captured in the NSDUH instrument. Second, the improvement in recall comes at the cost of reduced precision (35.9% for CSBT-Tuned versus 42.2% for CSBT-Untuned), meaning a larger proportion of positive predictions are false alarms. The acceptability of this trade-off depends on the downstream consequences of false positives in the specific deployment context. The use of hypothetical rather than actual clinical experts represents a methodological simplification appropriate for framework demonstration; however, the validity of the evaluation approach does not depend on these specific profiles because the weight bounds are derived from decision-analytic principles and the clinical characteristics of opioid misuse, and the sensitivity analysis confirms that model selection conclusions are robust across the full range of theoretically justified weights. Future applications of this framework should employ actual multidisciplinary expert panels, and the swing weight methodology presented here provides a structured protocol for such elicitation. Future work should also explore additional approaches to further improve minority class detection, including asymmetric cost functions with empirically calibrated cost ratios, ensemble methods combining multiple imbalance-handling strategies, and deep learning architectures capable of learning complex nonlinear relationships in high-dimensional survey data. Additionally, prospective validation studies would help establish the real-world clinical utility of these models for early identification and intervention in populations at risk for prescription opioid misuse. 4.3. Clinical Interpretability Through Detailed Model Diagnostics Beyond summary statistics, examining individual confusion matrix partitions and learning curve trajectories provides direct visibility into model error structures and operating point mechanics. While all three models achieved comparable scalar F1 scores, their underlying error distributions diverge substantially in ways that dictate clinical applicability. As established by the confusion matrix decomposition, CSBT-Tuned achieves an inverted error profile wherein positive detections exceed missed cases, whereas the untuned baseline concentrates errors as unflagged positive cases. Evaluating operating-point error distributions reveals how decision threshold calibration directly alters the clinical sensitivity-specificity trade-off, providing operational clarity that aggregate summary scores conceal. The learning curve analysis provides complementary diagnostic information regarding model stability and data efficiency. The convergence of training and validation curves observed for the CSBT models, wherein training accuracy decreased while validation accuracy increased until the curves met at approximately 80% of training data, indicates that these approaches transitioned from an overfitting regime at small sample sizes to stable generalization at larger sample sizes. The subsequent plateau in validation accuracy beyond 40% of training data suggests that performance limitations stem from model capacity or feature informativeness rather than insufficient training observations. In contrast, the continued improvement trajectory of RUSBoost learning curves at 100% data utilization indicates potential for enhanced performance with expanded datasets, an important consideration for longitudinal studies incorporating additional survey years. The decreasing training accuracy observed in the CSBT models warrants particular attention, as this pattern contradicts conventional expectations that training performance should increase or remain stable with additional data. This phenomenon arises from the cost-sensitive learning objective, wherein the model optimizes for weighted misclassification cost rather than raw accuracy. As training set size increases, the model encounters greater diversity in positive class presentations, requiring more conservative decision boundaries that reduce apparent training accuracy while improving generalization to the validation set. The simultaneous increase in validation accuracy confirms that this training accuracy reduction reflects improved generalization rather than degraded model quality. The combination of confusion matrix inspection and learning curve analysis reveals information inaccessible through aggregate metrics. Specifically, the observation that CSBT-Tuned achieves a true positive count exceeding its false negative count, combined with the convergence and plateau in its learning curves, suggests a model that has achieved clinically meaningful minority class detection at a stable operating point. Neither finding would be apparent from examining accuracy, precision, recall, or F1 scores in isolation. Healthcare applications require this level of diagnostic transparency to inform deployment decisions, threshold selection, and expectations regarding model behavior across patient populations. These findings support the broader recommendation that machine learning studies in clinical domains should routinely report complete confusion matrices and learning curves alongside aggregate performance metrics [22,28,36,40]. The convergence characteristics of learning curves indicate whether models have achieved stable performance or may benefit from additional data collection, while full confusion matrix breakdowns quantify the operational balance between diagnostic sensitivity and false alarm burden. 4.4. Feature Importance Interpretation and Model Selection Validity The clinical interpretability of feature importance analyses depends critically on the validity of the underlying model from which importance values are derived. This study demonstrates that feature importance extracted from a model selected solely on aggregate performance metrics may yield misleading clinical insights, as such models may fail to capture the relationships most relevant to minority class prediction. Had the CSBT-Untuned model been selected as the preferred classifier based on its superior overall accuracy (86.7% versus 83.6% for CSBT-Tuned), subsequent feature importance analysis would have been derived from a model that failed to detect 61.5% of individuals with documented opioid misuse history. Any clinical conclusions regarding risk factors for prescription opioid misuse drawn from such a model would be fundamentally compromised: the model’s feature importances would reflect predictors useful for classifying the majority (non-misuse) class rather than predictors that discriminate individuals at risk for the outcome of clinical interest. The CSBT-Tuned model, by contrast, was selected based on criteria directly aligned with clinical screening objectives: it achieved the only confusion matrix configuration where true positives exceeded false negatives, and its learning curves demonstrated stable convergence without evidence of continued improvement that would suggest underfitting. Feature importance values from this model therefore reflect predictors that contribute to identifying positive cases specifically, rather than predictors that merely maximize overall classification accuracy through majority class dominance. The prominence of lifetime alcohol use (alcever) and lifetime heroin use (herever) as top predictors aligns with clinical understanding of polysubstance use patterns and shared vulnerability factors across substance use disorders [41,42]. The importance of age category (catage) and marital status (irmarit) suggests that life stage transitions may represent periods of elevated risk warranting targeted screening. The contribution of major depressive episode (amdeyr) supports established evidence for mood disorder comorbidity in opioid misuse populations and suggests potential value in integrated mental health and substance use screening protocols [41,43,44,45,46]. These interpretations carry clinical validity precisely because they emerge from a model demonstrated to meaningfully detect the outcome of interest. Feature importance analyses from models with poor minority class performance, regardless of their aggregate accuracy, should be interpreted with substantial caution, as the identified “important” features may primarily explain majority class membership rather than the clinically relevant minority outcome [47]. This principle extends beyond the present application: any machine learning study seeking to derive clinical insights from feature importance should first establish that the source model achieves adequate detection of the outcome driving the clinical question, rather than relying on aggregate metrics that may obscure critical performance failures. 4.5. Clinical Evaluation Framework and Model Selection The application of the CDPS framework to the present study illustrates a principle central to clinically oriented evaluation: the identification of an optimal model depends on the clinical context in which that model will be deployed. Whereas standard machine learning evaluation treats performance metrics as fixed quantities and generic MCDM methods apply domain-agnostic aggregation rules, the proposed framework derives its composite metrics from clinically grounded parameterizations that directly reflect the consequences of diagnostic errors in medical screening contexts. The three models evaluated in this study achieved similar aggregate performance metrics, with F1 scores ranging from 40.28% to 42.86% and accuracy ranging from 83.6% to 86.7%. Traditional evaluation approaches that rely on such aggregate metrics would provide limited guidance for model selection. The CDPS framework, by contrast, reveals meaningful distinctions that emerge only when clinical priorities are explicitly incorporated into the evaluation. The resulting model rankings across the four primary evaluation frameworks are summarized in Table 30. The sensitivity priority weighting scheme (α2=0.60) represents the most clinically appropriate configuration for opioid misuse screening applications. In this context, the cost of failing to identify an individual with opioid misuse history substantially exceeds the cost of generating a false alarm. A missed case may result in continued substance misuse without intervention, whereas a false positive leads to additional assessment that ultimately confirms no intervention is indicated. This asymmetry, emphasized in the clinical evaluation literature, justifies weighting sensitivity more heavily than specificity for screening purposes. Under sensitivity priority weighting, CSBT-Tuned achieved a CDPS of 0.623, representing a 13.5% relative improvement over CSBT-Untuned (0.549). This improvement quantifies the clinical value of threshold optimization: by adjusting the decision threshold from the default 0.50 to the optimized value of 0.33, the CSBT-Tuned model detects 51.3% of positive cases compared to 38.5% for CSBT-Untuned. The CDPS framework successfully formalizes this clinical gain, whereas unweighted accuracy criteria incorrectly penalize the tuned model due to majority-class dominance. The convergence of CDPS values under specificity priority conditions (all models within 0.007) indicates that when false alarm minimization is the primary objective, the choice among these three models matters less. All three models achieve specificity exceeding 87%, and the modest differences in this metric do not translate into clinically meaningful distinctions when specificity is heavily weighted. This finding suggests that threshold optimization and class imbalance handling techniques provide their greatest value in screening contexts where sensitivity is prioritized. The present results also demonstrate the relationship between the CDPS framework and the model selection criteria employed in this study. The CSBT-Tuned model was identified as the preferred classifier based on two criteria: superior minority class detection (the only model achieving TP > FN on the test set) and stable learning curve convergence. The CDPS analysis confirms this selection by demonstrating that CSBT-Tuned achieves superior performance under the clinical weighting scheme most appropriate for screening applications. The concordance between these two evaluation approaches—one based on confusion matrix characteristics and learning dynamics, the other based on weighted composite metrics—strengthens confidence in the model selection decision. These findings carry implications for machine learning studies in clinical domains more broadly. Evaluation frameworks that treat all errors as interchangeable fail to capture the asymmetries inherent in medical decision making [1,22]. The CDPS and related clinically oriented metrics provide a principled approach for incorporating clinical judgment into model evaluation, ensuring that the selected model is not merely statistically optimal but clinically appropriate for its intended application. 4.6. Predictive Utility and the Sensitivity-PPV Trade-Off The CPUS analysis reveals a tension that the CDPS analysis alone does not capture: the trade-off between discriminative performance and predictive utility. CSBT-Tuned was identified as the preferred model based on CDPS analysis, achieving the highest score under sensitivity priority conditions due to its superior detection of positive cases. However, the same threshold adjustment that improved sensitivity also degraded PPV, resulting in inferior CPUS performance under conditions where the reliability of positive predictions is prioritized. This pattern has important implications for clinical deployment. A positive prediction from CSBT-Tuned is correct approximately 35.9% of the time, compared to 42.2% for CSBT-Untuned. In practical terms, approximately two-thirds of individuals flagged by CSBT-Tuned as having opioid misuse history do not, in fact, have such history. Whether this trade-off is acceptable depends on the consequences of false positive predictions in the specific deployment context. In screening applications where positive predictions trigger low-cost, non-invasive follow-up assessments, the reduction in PPV may be acceptable in exchange for improved case detection. The additional false positives represent individuals who receive unnecessary follow-up but are ultimately reassured that no intervention is indicated. In contexts where positive predictions trigger stigmatizing labels, employment consequences, or invasive interventions, the lower PPV of CSBT-Tuned may be problematic despite its superior sensitivity. The uniformly high NPV across all models (exceeding 0.92) reflects the influence of disease prevalence on predictive values. When prevalence is low, as in the present sample where approximately 11.6% of individuals have opioid misuse history, negative predictions are reliable regardless of model sensitivity because most individuals in the population are truly negative. This finding suggests that all three models can confidently rule out opioid misuse history when they predict its absence, with fewer than 8% of negative predictions being incorrect. The complementary perspectives provided by CDPS and CPUS underscore the value of clinically oriented evaluation frameworks. A complete assessment of model suitability requires consideration of both discriminative ability (how well does the model separate positive from negative cases?) and predictive utility (what does a positive or negative prediction mean for the individual patient?). Models that excel on one dimension may not excel on the other, and the appropriate balance depends on the clinical context and the relative costs of different error types. For opioid misuse screening, where identifying at-risk individuals for further assessment is the primary objective and false positives lead to additional evaluation rather than immediate intervention, the CDPS results favor CSBT-Tuned. In contexts where the consequences of false positive predictions are more severe, the CPUS results suggest that CSBT-Untuned may provide superior predictive utility despite its lower sensitivity. 4.7. Endpoint Accuracy and Class-Balanced Evaluation The WEAS analysis provides further evidence that aggregate performance metrics can obscure clinically meaningful distinctions between models. The complete reversal of model rankings between raw accuracy (CSBT-Untuned highest) and WEAS (CSBT-Tuned highest) demonstrates that model selection depends critically on whether the evaluation framework treats both outcome classes as equally important or allows majority class performance to dominate the assessment. For binary classification problems such as the present opioid misuse prediction task, the WEAS reduces to balanced accuracy, which assigns equal weight to correct classification of positive and negative cases regardless of their prevalence. This property is particularly relevant in clinical screening contexts where both types of classification errors carry meaningful consequences: false negatives represent missed opportunities for early intervention, whereas false positives result in unnecessary follow-up evaluation. The WEAS framework makes explicit the implicit assumption of class equality that raw accuracy obscures through its prevalence-weighted averaging. The gap between raw accuracy and WEAS observed in this study (ranging from 0.140 to 0.209 across models) reflects a general property of imbalanced classification: when one class substantially outnumbers the other, raw accuracy becomes insensitive to minority class performance. In the present dataset, where approximately 88% of observations belong to the negative class, a model that classified all observations as negative would achieve 88% raw accuracy while providing zero clinical utility for identifying individuals with opioid misuse history. The WEAS framework penalizes such degenerate solutions by treating per-class accuracies symmetrically. The WEAS framework offers additional value for potential extensions of the present work to multi-class ordinal outcomes. If opioid misuse were characterized along a severity continuum (for example: no misuse, experimental use, regular misuse, severe misuse, opioid use disorder), the full WEAS formulation would enable differential weighting of endpoint accuracy versus intermediate class accuracy. Clinical decision making in such contexts often depends primarily on distinguishing the endpoints: is the individual free of problematic use, or does the individual meet criteria for severe disorder requiring intensive intervention? Errors among intermediate categories, while undesirable, may carry less immediate clinical consequence than misclassifying an individual as healthy when severe disorder is present, or vice versa. The convergence of WEAS, CDPS under balanced weighting, and the AUC values (0.66 to 0.70 for CSBT-Tuned) reflects the mathematical relationships among these metrics when applied to binary classification. All three approaches share the property of treating both classes symmetrically, and all three identify CSBT-Tuned as the preferred model. This convergence strengthens confidence in the model selection decision by demonstrating that the conclusion is robust across multiple evaluation frameworks that share the common assumption of equal class importance. 4.8. Balancing Accuracy and Error Minimization The CEPM analysis reveals a fundamental tension in classifier evaluation that neither raw accuracy nor balanced accuracy alone captures: the trade-off between minimizing total errors and achieving balanced per-class performance [2]. CSBT-Untuned achieved the lowest total error rate (13.3%) but the worst balanced accuracy (65.8%), whereas CSBT-Tuned achieved the highest balanced accuracy (69.6%) but the worst error rate (16.4%). The CEPM framework, through its balancing parameter λ, provides a principled mechanism for navigating this trade-off according to clinical priorities. The crossover point at λ≈0.39 carries direct clinical interpretation. Below this threshold, the evaluation framework values total error minimization more heavily than balanced per-class performance. In such contexts, a model that makes fewer total errors is preferred even if those errors are concentrated among the minority class. Above this threshold, the framework values balanced per-class performance more heavily, and a model that detects minority class cases more reliably is preferred even if it makes more total errors. For opioid misuse screening applications, the appropriate value of λ depends on the relative costs of the two component metrics. If resources for follow-up assessment are severely constrained and each positive prediction triggers substantial downstream costs, lower values of λ may be appropriate, favoring models that minimize false alarms even at the cost of reduced sensitivity. If the primary objective is early identification of individuals at risk, with follow-up assessment being relatively low-cost, higher values of λ are appropriate, favoring models that achieve balanced detection of both classes. The adaptation of CEPM to binary classification warrants methodological comment. In the original multi-class formulation, the distance penalty differentiates between errors of varying severity: confusing healthy with severely diseased (distance 4) incurs greater penalty than confusing healthy with mildly affected (distance 1). In binary classification, this severity differentiation is absent because all errors have unit distance. The penalty therefore reduces to the overall error rate, and the CEPM becomes a weighted combination of balanced accuracy and raw accuracy. This mathematical simplification preserves the utility of the λ parameter for balancing competing objectives but loses the severity-weighting property that motivates the CEPM in multi-class contexts. Future extensions of this work to ordinal severity outcomes, such as a graded scale from no misuse through mild, moderate, and severe misuse, would restore the full functionality of the CEPM framework. In such settings, the penalty term would differentiate between misclassifications of varying clinical severity, penalizing confusion between no misuse and severe misuse more heavily than confusion between adjacent severity grades. This property would align model evaluation with clinical decision making, where distinguishing extreme categories typically carries greater consequence than distinguishing intermediate categories. The convergence of evaluation frameworks observed in this study, with CDPS, WEAS, and CEPM at high λ all favoring CSBT-Tuned, provides robust support for model selection when balanced per-class performance is prioritized. The divergence at low λ values, where CEPM favors CSBT-Untuned, highlights that this consensus depends on the assumption that both classes merit equal consideration. When total error minimization takes precedence, as might occur in resource-constrained settings, the model selection calculus changes accordingly. 4.9. Clinical Implications of Penalty Matrix and Crossover Analysis The custom penalty matrix framework provides a principled approach to translating clinical cost considerations into quantitative model selection criteria. Our crossover analysis identified cFN*=3.03 as the critical threshold at which CSBT-Tuned becomes cost-optimal relative to CSBT-Untuned. This value has direct clinical interpretation: when the clinical value of preventing one missed case of opioid misuse is worth accepting approximately three false alarms, the threshold-optimized model achieves lower total expected cost despite its higher raw error rate. The clinical plausibility of this threshold depends on the specific screening context. For instance, routine screening with low-cost follow-up procedures may warrant penalty ratios near 2:1, whereas high-risk populations and overdose prevention contexts justify ratios of 5:1 or higher, making CSBT-Tuned the clearly superior choice. Figure 30 maps these clinical contexts to approximate penalty ratios based on downstream intervention characteristics. Sensitivity and Practical Implementation The narrowness of cost differences at intermediate penalty ratios, such as the 2.4% difference observed under a 3:1 ratio, suggests that in contexts where the appropriate ratio is uncertain, model selection has a limited impact on total expected cost. However, the cost differences amplify substantially at extreme penalty ratios, where selecting the wrong model could increase expected costs by over 13%. Table 31 presents the sensitivity of optimal model selection to penalty ratio uncertainty across different screening contexts. To guide practitioners, we propose the decision flowchart in Figure 31, which bases penalty selection on population risk and intervention characteristics. While the penalty matrix framework assumes linear costs and contains inherent uncertainties near crossover thresholds, it remains a transparent mechanism for ensuring that model selection trade-offs align with institutional priorities and ethical commitments. 4.10. Decision-Analytic Interpretation of Model Performance 4.10.1. Reconciling Expected Utility and Clinical Composite Metrics The expected utility analysis revealed an apparent paradox: under most utility parameterizations, CSBT-Untuned achieved higher expected utility than CSBT-Tuned, despite CSBT-Tuned’s superior performance on sensitivity-prioritized composite metrics such as CDPS. This divergence warrants careful interpretation, as it illuminates fundamental differences between metric-based and utility-based evaluation frameworks. The explanation lies in the interaction between prevalence and the probability weights in the expected utility calculation. At the observed prevalence of 11.7%, true negative outcomes dominate the probability distribution. For CSBT-Untuned, P(TN)=0.822 compared to P(TN)=0.777 for CSBT-Tuned. This 4.5 percentage point difference in true negative probability, when multiplied by the positive utility of correct negative classification (UTN=0.80 to 0.90), generates substantial utility advantage for CSBT-Untuned that offsets its inferior performance on the minority positive class. Figure 32 presents a decomposition of expected utility contributions that clarifies this relationship. This finding has important implications for model selection in low-prevalence screening contexts. Composite metrics such as CDPS with sensitivity-priority weighting implicitly assume that improvements in sensitivity carry weight proportional to the specified α2 coefficient, regardless of prevalence. In contrast, expected utility calculations naturally incorporate prevalence through the probability weights, such that sensitivity improvements have diminishing marginal utility as prevalence decreases. The practical implication is that expected utility analysis may recommend different models than composite metric analysis, even when both frameworks incorporate the same clinical priorities. This divergence is not a failure of either approach but rather reflects their different scopes: composite metrics evaluate discriminative performance independent of prevalence, while expected utility evaluates total clinical value in a specific population context. Table 32 summarizes the model rankings under composite metrics versus expected utility. 4.10.2. Clinical Interpretation of Net Benefit Results The decision curve analysis provided actionable guidance for model selection based on the treatment threshold that characterizes a clinical context. The treatment threshold pt represents the probability of disease at which a clinician would be indifferent between intervening and not intervening. This threshold implicitly encodes the ratio of harms from unnecessary intervention to harms from missed disease:pt=HarmFPHarmFP+HarmFN At pt=0.10, the implied harm ratio is HarmFN/HarmFP=9. At pt=0.25, the implied ratio is 3. At pt=0.50, the implied ratio is 1 (equal harms). Figure 33 maps clinical contexts to approximate treatment thresholds and the corresponding optimal model. The crossover between CSBT-Tuned and RUSBoost occurred at approximately pt=0.18. Below this threshold, CSBT-Tuned’s superior true positive detection outweighs its higher false positive rate. Above this threshold, RUSBoost’s more balanced performance achieves higher net benefit. CSBT-Untuned achieved the lowest net benefit among classifiers across most of the threshold range, reflecting the cost of its low sensitivity even when specificity is highly valued. The range over which each classifier achieved positive net benefit (superior to “Treat None”) provides a measure of clinical robustness. CSBT-Tuned maintained positive net benefit for pt∈[0.05,0.37]; RUSBoost maintained positive net benefit for pt∈[0.05,0.47]; CSBT-Untuned maintained positive net benefit for pt∈[0.05,0.45]. RUSBoost thus demonstrated the widest range of clinical utility, though CSBT-Tuned achieved higher peak net benefit at low thresholds. 4.10.3. Cost-Effectiveness Implications for Implementation The cost-effectiveness analysis demonstrated that all classifier-guided screening strategies dominated the no-screening alternative, achieving both lower costs and better health outcomes. This finding supports the economic viability of implementing machine learning-based opioid misuse screening in healthcare settings. The incremental analysis revealed a cost-effectiveness frontier connecting CSBT-Untuned, RUSBoost, and CSBT-Tuned in order of increasing cost and effectiveness. The ICERs along this frontier ($6463/QALY for RUSBoost relative to CSBT-Untuned; $13,840/QALY for CSBT-Tuned relative to RUSBoost) fall well below conventional willingness-to-pay thresholds, indicating that healthcare systems should be willing to incur the additional costs associated with higher-sensitivity screening. The probabilistic sensitivity analysis revealed important uncertainty in these conclusions. At low willingness-to-pay thresholds (<$15,000/QALY), CSBT-Untuned had a 58% probability of being the most cost-effective option, reflecting its lower screening and follow-up costs. At high willingness-to-pay thresholds (>$50,000/QALY), CSBT-Tuned had a 56% probability of being most cost-effective, reflecting the value placed on avoiding false negative harms. Table 33 synthesizes the decision-analytic findings into implementation recommendations. 4.10.4. Limitations of the Decision-Analytic Framework Several limitations of the decision-analytic evaluation warrant acknowledgment. First, the utility and cost parameters were derived from literature estimates and expert judgment rather than primary data collection. The sensitivity of conclusions to parameter uncertainty was addressed through probabilistic analysis, but structural uncertainty regarding the appropriate parameter values remains. Second, the analysis assumed a one-year time horizon and did not model long-term outcomes of early intervention versus missed detection. Opioid misuse trajectories evolve over years, and the full value of early detection may not be captured in short-term QALY estimates. Third, the expected utility framework assumes that utilities are linear and additive across outcomes. In practice, decision makers may exhibit risk aversion or other non-linear preferences that would alter optimal model selection [48]. Fourth, the net benefit formulation assumes that the treatment threshold is known and stable across patients. In practice, treatment decisions must account for the fact that machine learning classifiers are fundamentally statistical rather than confirmatory. These tools identify conditions based on how a patient fits into broad, nationwide statistical patterns; however, there will always be specific patients who cannot be accurately described by those statistics. Relying on such a tool for a single patient is akin to prescribing treatment for condition A simply because “most people” in that statistical bucket have it, potentially ignoring alternative explanations for that specific individual’s symptoms. This inherent limitation underscores the necessity of personalized clinical judgment and the multidimensional validation described in this paper before any model is used to guide treatment for new, unseen patients. Despite these limitations, the decision-analytic framework provides a principled approach to connecting classifier performance metrics to downstream clinical and economic outcomes. The convergence of multiple analytic approaches (expected utility, net benefit, cost-effectiveness) on the recommendation of CSBT-Tuned for high-sensitivity screening and RUSBoost for balanced applications provides robust support for these implementation recommendations. 4.11. Integrating Multiple Clinical Perspectives The application of four complementary evaluation metrics to the present classification problem reveals a fundamental principle of clinically oriented model assessment: different metrics capture different aspects of clinical utility, and disagreements between metrics carry substantive meaning. A model that excels on discriminative metrics (CDPS, WEAS) may not excel on predictive utility metrics (CPUS), and vice versa. These disagreements are not failures of the metrics but rather reflections of the multi-dimensional nature of clinical performance. Integrating CDPS, CPUS, WEAS, and CEPM demonstrates how complementary performance indices expose distinct operational trade-offs. CSBT-Tuned excels on discriminative and class-balanced dimensions (CDPS, WEAS) by rewarding minority class case detection. Conversely, CSBT-Untuned preserves higher positive predictive utility under PPV-priority constraints, and CEPM establishes an explicit crossover (λ≈0.39) where preference shifts from total error minimization to balanced class accuracy. For opioid misuse screening applications, the integrated metric analysis supports the selection of CSBT-Tuned as the preferred model under the following conditions: (1) the primary objective is identifying individuals at risk for opioid misuse, accepting higher false positive rates in exchange for improved case detection; (2) both outcome classes are considered equally important for evaluation purposes, rather than allowing majority class performance to dominate; and (3) the clinical context can accommodate the lower positive predictive value (35.9%) associated with threshold optimization. Under these conditions, CSBT-Tuned achieves the highest point scores among viable classifiers across CDPS, WEAS, and CEPM (at λ>0.39). Although RUSBoost achieved competitive point estimates on specific composite scores, it was disqualified from final model selection due to non-convergent learning dynamics, establishing CSBT-Tuned as the top-performing viable classifier. The exclusion of RUSBoost from model selection merits explicit discussion. Despite achieving the highest CEPM at λ=0.5 (0.271) and competitive performance across other metrics, RUSBoost exhibited learning curves that neither converged nor plateaued during training. This behavior suggests that the model’s performance on the test set may not be reproducible and that the model may be sensitive to the particular random samples encountered during training. Clinical deployment requires models with stable, predictable behavior; a model whose learning dynamics indicate ongoing instability cannot be recommended regardless of its metric performance on a single test set evaluation. Conversely, if the clinical context prioritizes the reliability of positive predictions, such that each positive screen triggers resource-intensive follow-up, the integrated analysis supports consideration of CSBT-Untuned. This model achieves the highest CPUS under PPV-priority weighting and the highest CEPM at low λ values, reflecting its conservative classification strategy that produces fewer false alarms at the cost of reduced sensitivity. Figure 34 operationalizes these findings as a decision framework for model selection. The framework maps clinical questions to specific model recommendations between the two viable candidates: if case detection is paramount and false positives are tolerable, CSBT-Tuned is preferred; if prediction reliability is paramount and false negatives are tolerable, CSBT-Untuned is preferred. The convergence of CDPS (under balanced weighting), WEAS, and lower-bound AUC estimates at approximately 0.66 to 0.70 for all three models deserves methodological comment. This convergence is not coincidental but reflects mathematical relationships among these metrics when applied to binary classification with AUC estimated as the average of sensitivity and specificity. All three formulations reduce to weighted combinations of sensitivity and specificity, and when weights are equal, they produce identical rankings. This convergence provides robustness to the finding that CSBT-Tuned achieves superior balanced performance: the conclusion does not depend on which specific metric is chosen, provided that metric treats both classes symmetrically. The integrated metric framework demonstrates that comprehensive model evaluation requires explicit consideration of both clinical priorities and model stability. Reporting a single metric, whether accuracy, F1 score, or AUC, obscures the multi-dimensional nature of clinical performance and may lead to model selections that are statistically optimal but clinically suboptimal [2,49,50,51]. Furthermore, evaluating learning dynamics alongside composite metrics provides an important safeguard: non-convergent models such as RUSBoost can be identified and disqualified prior to deployment, ensuring that candidate models exhibit both statistical utility and learning stability. 4.12. Systematic Weight Elicitation and the Defensibility of Model Selection The demonstration of swing weight elicitation methodology to derive CDPS weighting coefficients addresses a fundamental criticism of composite evaluation metrics: that weights are often chosen arbitrarily or, worse, adjusted post hoc to favor a preferred model. This structured elicitation demonstration, utilizing five hypothetical clinical expert profiles representing diverse stakeholder perspectives, produced documented and defensible weight selections grounded in explicit clinical reasoning rather than researcher preference. The elicited weights revealed systematic differences in clinical priorities across stakeholder groups that would not have been apparent from predefined weighting scenarios alone. Clinicians and patient advocates consistently prioritized sensitivity, reflecting concern about missing individuals who could benefit from early intervention. The hypothetical addiction medicine profile was assigned the highest raw importance score (100) for sensitivity improvement, yielding a normalized weight of α2=0.53. The hypothetical patient advocate profile was modeled with an even stronger preference, with α2=0.61 after normalization. These priorities align with the ethical imperative in screening applications to minimize false negatives when the cost of missed cases includes preventable morbidity. In contrast, the hypothetical hospital quality officer profile prioritized specificity (α3=0.49) to reflect simulated institutional concerns about resource allocation and the downstream costs of false positive follow-up. The epidemiologist uniquely prioritized AUC (α1=0.42) as a threshold-independent measure of overall discriminative ability, reflecting a methodological perspective that values model performance across all possible operating points. These divergent priorities are not contradictions to be resolved but rather legitimate perspectives reflecting different stakeholder roles in healthcare delivery. The convergence of elicited weights between the predefined balanced scenario (α2=0.33) and sensitivity priority scenario (α2=0.60) carries methodological significance. The simple average of expert weights (α2=0.44) and clinically weighted average (α2=0.52) both assign greater importance to sensitivity than the balanced scenario but less than the extreme sensitivity priority scenario. This finding suggests that the predefined sensitivity priority scenario may slightly overweight sensitivity relative to expert consensus, while the balanced scenario underweights it. The elicited weights thus provide empirically grounded alternatives to predefined scenarios that may better reflect actual clinical priorities for opioid misuse screening. The preservation of model rankings across all elicitation-based aggregation strategies strengthens confidence in the selection of CSBT-Tuned as the preferred classifier. Under simple averaging, CSBT-Tuned achieved CDPS of 0.672 compared to 0.660 for RUSBoost and 0.622 for CSBT-Untuned. Under clinical weighting, CSBT-Tuned achieved 0.659 compared to 0.639 and 0.596 respectively. The consistency of these rankings across aggregation methods indicates that the superiority of CSBT-Tuned is not an artifact of particular weight choices but reflects genuine clinical advantage for screening applications where expert priorities are systematically incorporated. The documentation of expert rationales alongside numerical weights provides an audit trail that enhances the reproducibility and credibility of the evaluation. Future researchers can assess whether the elicitation process was appropriate for their specific clinical context, can replicate the evaluation with different expert panels, and can understand the clinical reasoning underlying particular weight selections. This transparency addresses concerns about the subjectivity of composite metrics by making the subjective elements explicit and traceable rather than hidden within arbitrary weight choices. The disagreement among experts, rather than being a limitation, provides valuable information about the uncertainty surrounding appropriate weights. The range of sensitivity weights from 0.29 to 0.61 across the five experts spans nearly the entire plausible range, indicating genuine uncertainty about how heavily sensitivity should be weighted relative to other metrics. This disagreement motivates the sensitivity analysis reported in the results, which demonstrates that model rankings are robust across the full range of expert opinion rather than dependent on particular weight values within that range. For practitioners applying this framework in other clinical contexts, we recommend the following approach to weight justification, illustrated schematically in Figure 35. First, establish theoretical bounds based on the screening-versus-confirmation distinction and the relative severity of false negative versus false positive consequences for the specific condition under study. Second, conduct structured elicitation with relevant stakeholders using swing weight or similar methodology. Third, verify that elicited weights fall within theoretical bounds; if elicited weights fall outside the justified range, the elicitation process should be revisited to reconcile stakeholder judgments with clinical constraints. Fourth, perform sensitivity analysis across the range of plausible weights to assess robustness of model selection conclusions. This multi-source justification strategy reduces dependence on any single weight derivation method while ensuring that the final weights reflect both clinical reasoning and stakeholder priorities. 4.13. Limitations of Weight Elicitation in the Present Study Several limitations of the weight elicitation process warrant acknowledgment. The expert panel of five members, while representing diverse stakeholder perspectives, constitutes a small sample from which to generalize about clinical priorities. A larger panel might reveal additional perspectives or produce different aggregated weights. The selection of experts was purposive rather than random, potentially introducing selection bias if the chosen experts hold views unrepresentative of their broader professional communities. The swing weight methodology, while structured, still requires experts to make difficult comparative judgments about the value of metric improvements. Some experts may find such comparisons counterintuitive or may interpret the elicitation questions differently, introducing measurement error into the weight estimates. The use of worst and best plausible bounds, while grounded in observed performance ranges, involves subjective judgment that affects the resulting weights. The elicitation was conducted for a single clinical application (opioid misuse screening) and may not generalize to other screening contexts. The relative priorities of sensitivity, specificity, and AUC may differ substantially for screening applications with different prevalence rates, different consequences of false positives and false negatives, or different downstream interventions triggered by positive predictions. The weights elicited in this study should be interpreted as context-specific rather than universally applicable. Despite these limitations, the systematic elicitation process represents a substantial improvement over arbitrary weight selection or post hoc adjustment. The documented disagreement among experts, the preservation of model rankings across aggregation methods, and the alignment of elicited weights with predefined clinical scenarios all support the validity of the weight selection process. Future work should explore larger expert panels, alternative elicitation methodologies, and cross-context validation of elicited weights to further strengthen the defensibility of composite metric evaluation. 4.14. Uncertainty Quantification and the Precision of Performance Claims The bootstrap confidence intervals reported in this study transform the interpretation of model performance from point estimates conveying false precision to interval estimates that honestly represent the state of knowledge given the available data. The observation that CSBT-Tuned achieved sensitivity of 0.513 becomes more informative when accompanied by the 95% confidence interval [0.447, 0.580], indicating that the true sensitivity plausibly ranges from approximately 45% to 58% given sampling variability. The asymmetric precision of component metrics carries important implications for model evaluation. The standard error for sensitivity (0.032 to 0.034 across models) was approximately five times larger than for specificity (0.006 to 0.009), reflecting the smaller effective sample size for the positive class (n = 226) compared to the negative class (n = 1711). This asymmetry means that sensitivity estimates carry substantially greater uncertainty than specificity estimates, a consideration that should inform the confidence placed in sensitivity-based model comparisons. Claims about sensitivity differences between models require more caution than claims about specificity differences because the former are estimated with lower precision. The near-overlap of sensitivity confidence intervals between CSBT-Untuned [0.323, 0.451] and CSBT-Tuned [0.447, 0.580] provides statistical context for the observed point estimate difference (0.513 versus 0.385). The lower bound of the CSBT-Tuned interval (0.447) falls only 0.004 below the upper bound of the CSBT-Untuned interval (0.451), suggesting that the sensitivity improvement is statistically significant at approximately the 95% level but not with overwhelming confidence. This finding tempers the interpretation of the 33% relative improvement in sensitivity: while the point estimate suggests substantial clinical benefit from threshold optimization, the confidence intervals indicate that the true improvement could be smaller than the point estimate suggests. The substantial overlap of sensitivity confidence intervals between CSBT-Tuned [0.447, 0.580] and RUSBoost [0.412, 0.544] indicates that the observed difference (0.513 versus 0.478) may not be statistically reliable. The true sensitivities of these two models could be equal or even reversed from the observed point estimates. This finding suggests that the preference for CSBT-Tuned over RUSBoost, while supported by point estimates, should be held with less confidence than the preference for CSBT-Tuned over CSBT-Untuned. The propagation of uncertainty through composite scores reveals that CDPS confidence intervals inherit the uncertainty of their component metrics. Under sensitivity priority weighting, the CSBT-Tuned interval [0.579, 0.668] does not overlap with the CSBT-Untuned interval [0.509, 0.590], confirming that the CDPS improvement is statistically significant at the 95% level. This finding provides stronger evidence for the clinical value of threshold optimization than the point estimates alone, as it demonstrates that the observed CDPS difference exceeds what would be expected from sampling variability. The convergence of CDPS confidence intervals under specificity priority weighting, where all three models achieved intervals spanning approximately 0.75 to 0.80, indicates that model selection under this scenario cannot be justified on statistical grounds. The observed differences in CDPS point estimates (0.768 to 0.775) fall well within the range of sampling variability. When specificity is heavily weighted, the choice among these three models should be based on secondary considerations such as interpretability, computational efficiency, or robustness to distributional shift rather than on CDPS values that are statistically indistinguishable. 4.15. Robustness of Model Rankings Across Weight Uncertainty The sensitivity analysis across the range of expert-elicited weights demonstrates that model rankings are robust to uncertainty in weight specification. CSBT-Tuned maintained the highest CDPS throughout the entire range of sensitivity weights from 0.29 to 0.61, with its advantage over CSBT-Untuned increasing from 2.6% at low sensitivity weights to 13.1% at high sensitivity weights. This robustness provides confidence that the selection of CSBT-Tuned is not an artifact of particular weight choices but reflects genuine clinical superiority across the range of priorities expressed by clinical experts. The increasing relative advantage of CSBT-Tuned at higher sensitivity weights aligns with clinical intuition. As screening applications place greater emphasis on case detection, the superior sensitivity of the threshold-optimized model provides greater clinical value. The 13.1% relative advantage at α2=0.61 (the maximum expert weight for sensitivity) quantifies the clinical benefit of threshold optimization for stakeholders who most heavily prioritize sensitivity, such as patient advocates and addiction medicine specialists. The preservation of model ranking order (CSBT-Tuned, RUSBoost, CSBT-Untuned) across all weight values within the expert range indicates that disagreement among experts about appropriate weights does not translate into disagreement about which model is preferred. This concordance strengthens the model selection decision: even though experts disagree about how heavily to weight sensitivity relative to other metrics, they would all prefer CSBT-Tuned regardless of which expert’s weights were adopted. The convergence of all models toward similar CDPS values at the lower end of the sensitivity weight range (α2=0.29) suggests that expert disagreement about model selection increases at lower sensitivity weights. When sensitivity is weighted minimally, the differences between models become smaller and may fall within the range of statistical uncertainty. This finding implies that the strength of the case for CSBT-Tuned depends on how heavily the clinical context values sensitivity: in contexts where sensitivity is paramount, CSBT-Tuned is clearly preferred; in contexts where sensitivity is less important, the model selection decision is less clear-cut. 4.16. Implications for Reporting and Interpretation The uncertainty quantification methodology employed in this study supports several recommendations for reporting machine learning results in clinical applications. First, confidence intervals should accompany all point estimates for performance metrics, enabling readers to assess whether observed differences between models exceed sampling variability. Second, when comparing models, explicit statements about whether confidence intervals overlap should supplement claims about point estimate differences. Third, sensitivity analysis across plausible weight ranges should be reported for composite metrics to reveal whether conclusions are robust or sensitive to weight specification. The pattern of results in this study illustrates how uncertainty-aware evaluation can inform model selection decisions. The statistically significant CDPS improvement of CSBT-Tuned over CSBT-Untuned under sensitivity priority weighting supports a confident selection decision. The statistically indistinguishable CDPS values under specificity priority weighting suggest that model selection in that context should be based on considerations other than CDPS. The substantial overlap of confidence intervals between CSBT-Tuned and RUSBoost across most scenarios indicates that the observed superiority of CSBT-Tuned may partially reflect sampling variability, warranting cautious interpretation of the preference ranking. These findings also carry implications for sample size planning in future studies. The relatively wide confidence intervals for sensitivity-related metrics (reflecting the smaller positive class sample size) suggest that studies seeking to detect sensitivity differences between models require larger samples of the positive class than might be inferred from overall sample size. For rare outcomes such as opioid misuse, this consideration may motivate oversampling strategies, case-control designs, or multi-year data collection to achieve adequate precision for clinically meaningful comparisons. 4.17. Integration of Extended Methodological Framework The systematic weight elicitation and uncertainty quantification extensions applied in this study address two distinct but related concerns about composite metric evaluation: the subjectivity of weight selection and the precision of performance claims. Together, these extensions strengthen the defensibility of model selection decisions by grounding weights in documented clinical judgment and by quantifying the statistical reliability of observed performance differences. Triangulating expert swing-weight elicitation, Monte Carlo parameter sensitivity, and bootstrap resampling confirms that CSBT-Tuned remains the robust choice for screening workflows while transparently bounding claims where sampling confidence intervals overlap (e.g., relative to RUSBoost). This joint framework establishes a defensible protocol for validating clinical classifiers under realistic deployment constraints. The methodological extensions employed in this study are not specific to the opioid misuse screening application but generalize to any classification problem where composite metrics guide model evaluation. The swing weight elicitation procedure can be applied whenever clinical experts are available to express priorities among competing performance dimensions. The bootstrap confidence interval methodology can be applied whenever a held-out validation set is available for model evaluation. The sensitivity analysis framework can be applied whenever weights carry uncertainty from expert disagreement or imprecise elicitation. Together, these tools provide a comprehensive methodology for clinically oriented model evaluation that acknowledges and addresses the subjective and statistical uncertainties inherent in performance assessment. Future methodological development should explore the integration of these extensions with formal decision analysis frameworks that incorporate the costs and benefits of different clinical outcomes. The weights elicited from experts reflect ordinal preferences about metric importance but do not directly encode the monetary or health-related costs of different error types. A full decision-analytic framework would combine the composite metric evaluation with explicit cost modeling, enabling model selection decisions that maximize expected utility rather than weighted performance scores. Such integration would further strengthen the clinical relevance of machine learning evaluation in healthcare applications. 5. Conclusions In machine learning classification, it is possible to achieve high accuracy that is nonetheless clinically useless. When applying task-specific AI tools to human lives, simple metrics are fundamentally insufficient. Impressive accuracy claims often mask algorithmic shortcuts that collapse the moment the AI encounters actual patients. As physicians who directly alter patients’ lives, clinicians cannot outsource their professional judgment to marketing materials. High aggregate metrics frequently result from improper training techniques, such as those that permit overfitting, numerous types of data leakage, or the exploitation of class imbalances. Beyond these technical risks, practitioners must remain mindful that machine learning models (similar to large language models) are fundamentally statistical tools. They are designed to work for patients who fit the training statistics, but they cannot be expected to serve as confirmatory evidence because specific individuals will always fall outside the bounds of aggregate data. Using these tools in clinical practice is comparable to diagnosing a patient based on nationwide statistics: assuming that because a condition is most common among similar profiles, it must apply to the patient in front of the clinician. Such logic can lead to prescribing treatment for condition A even when a patient’s unique situation might be better explained by other conditions. Therefore, performance claims should not be accepted unless they are accompanied by the comprehensive technical and clinically oriented analyses described in this paper. When accepting to work with AI tools, physicians should demand a report similarly detailed to this manuscript, keeping this study as guidance for what an evidence report must look like before making the decision to use AI tools on their patients.