The anesthesiologist’s guide to critically assessing machine learning research: a narrative review Authors: Felipe Ocampo Osorio, Sergio Alzate-Ricaurte, Tomas Eduardo Mejia Vallecilla, Gustavo Adolfo Cruz-Suarez Published: 2024 ABSTRACT AbstractArtificial Intelligence (AI), especially Machine Learning (ML), has developed systems capable of performing tasks that require human intelligence. In anesthesiology and other medical fields, AI applications can improve the precision and efficiency of daily clinical practice, and can also facilitate a personalized approach to patient care, which can lead to improved outcomes and quality of care. ML has been successfully applied in various settings of daily anesthesiology practice, such as predicting acute kidney injury, optimizing anesthetic doses, and managing postoperative nausea and vomiting. The critical evaluation of ML models in healthcare is crucial to assess their validity, safety, and clinical applicability. Evaluation metrics allow an objective statistical assessment of model performance. Tools such as Shapley Values (SHAP) help interpret how individual variables contribute to model predictions. Transparency in reporting is key in maintaining trust in these technologies and to ensure their use follows ethical principles, aiming to reduce safety concerns while also benefiting patients. Understanding evaluation metrics is essential, as they provide detailed information on model performance and their ability to discriminate between individual class rates. This article offers a comprehensive framework in assessing the validity, applicability, and limitations of models, guiding responsible and effective integration of ML technologies into clinical practice. A balance between innovation, patient safety and ethical considerations must be pursued.Keywords: Artificial intelligence, Machine learning, Clinical decision support system, Patient-specific modeling, Anesthesiology Introduction/Background Artificial Intelligence (AI) is a field of computer science which develops systems capable of performing tasks that traditionally have required human intelligence, such as pattern recognition, decision making, complex problem resolution, and natural language processing [1]. AI has evolved significantly, initially being a theoretical concept and now an essential technology to multiple disciplines, including medicine [2]. Among its applications are the development of algorithms to predict mortality in ICU patients [3], the early detection of the need for invasive mechanical ventilation in neonatal patients [4], the prediction of drug-induced toxicity [5], and the assessment of lung cancer risk from computed tomography scans [6]. It plays a role in improving diagnostics, optimizing treatments, and supporting clinical decision-making by analyzing large datasets and learning from them at a rapid rate [7]. Machine Learning (ML), a branch of AI, is used to develop systems that utilize historical datasets to generate predictions or make decisions without requiring explicit programming for individual tasks [8]. Deep Learning (DL), a subset of ML, uses complex algorithms based on neural networks to process and analyze large volumes of data, requiring greater complexity [9]. ML is divided into three main types: supervised, in which algorithms learn from labeled data and make predictions; unsupervised, in which hidden patterns are identified in unlabeled data, which are then used for clustering and anomaly detection; and reinforcement learning, where systems learn from feedback systems based on rewards and penalties from their own previous actions, while interacting with a changing environment [10]. A key distinction between DL and other types of ML lies in the type of data they are best suited to handle. Traditional ML models generally perform better with structured data, such as tables with clearly defined variables organized into rows and columns. In contrast, DL excels at processing unstructured data, such as images, text, and audio, by automatically extracting relevant features and patterns directly from raw data. This capability makes DL particularly useful in fields like medical imaging and natural language processing, where data does not follow a predefined structure [11]. ML has shown promising results in the field of anesthesiology, with several algorithms being prospectively validated and implemented in clinical settings. For example, ML algorithms have been successfully applied to predict intraoperative hypotension [12], forecast postoperative acute kidney injury [13], assess the risk of complications after hip surgery [14], and determine the ASA physical status classification (ASA-PS) using data from electronic health records [15]. The use of ML enhances precision and efficiency in practice and allows for personalized care, contributing to better clinical outcomes and quality of care. The accurate interpretation of ML models in healthcare carries significant responsibility, as their validity must be thoroughly evaluated beforehand to ensure patient safety [16]. To address this need, this article aims to provide a step-by-step guide for critically appraising supervised ML models in a medical context, helping anesthesiologists evaluate their validity, applicability, and limitations before models are implemented into clinical practice. Generalities Specification Defining the clinical use case for a machine learning (ML) model must precede data preprocessing. This specification directly governs data availability during the inference phase. The model should incorporate only those variables accessible at the point of inference, making feature selection a critical consideration. Rigorous evaluation requires identifying any features inappropriately included due to temporal constraints. In anesthesiology, for instance, variables like intraoperative urine output and blood loss documentation occur post-procedure, rendering them unsuitable for preoperative or intraoperative predictive modeling. Data preprocessing and data quality Proper data preprocessing serves as a fundamental requirement for ML model implementation. Inadequate preparation of data can result in outright errors during model execution and undermines the reliability of results [17]. This process involves detecting outliers (numeric data that significantly deviates from the rest and can skew results) and managing null or missing data. Several techniques can be applied to handle missing data, depending on the context. For time series data, researchers employ the last-value-carried-forward approach. Mean or median imputation serves continuous features, while modal imputation is suitable for categorical features. Missing indicators provide additional value for both types of features when the absence of data itself carries meaningful information [18]. It is also important to identify duplicate records, which can bias the analysis by overrepresenting certain data points [19]. Missing data in anesthesiology often stems from back-charting during critical patient interventions, compromising real-time data availability when predictions are most crucial. Feature normalization represents a critical preprocessing step that equalizes the contribution of continuous variables to model predictions. Continuous variables within datasets frequently span disparate scales and ranges, leading to disproportionate model influence from features with larger values or broader ranges. Implementing normalization techniques, such as min-max scaling [0,1] or z-score normalization (mean = 0, standard deviation = 1), standardizes numeric variables to a common scale. This standardization enhances model convergence and optimizes performance, particularly for scale-sensitive algorithms [20]. Relying on high-quality data is essential, as computational models need consistent and complete information to make reliable predictions. In contrast, low-quality data can lead to inaccurate or misleading results, compromising clinical decision-making and patient safety. Proper preprocessing ensures that models work with accurate information, thereby improving their ability to predict outcomes and personalize treatments effectively [21]. Machine learning models In ML, two classes of models are generally used and selection is based on the individual tasks they are created to perform: classification and regression models [22]. (Fig. 1). Classification models synthesize multiple input variables to assign data points to predefined outcome categories. In clinical applications, these models generate binary outcomes (presence/absence) for specific conditions based on patient characteristics. Some common subtypes of classification models include K-Nearest Neighbors (KNN), Decision Tree, and Support Vector Machine (SVM) [23, 24]. KNN uses distances to classify data based on proximity to labeled points, while SVM maximizes class separation by utilizing kernel transformed spaces to find optimal decision boundaries [25]. Tree based models use hierarchical structures of nodes and branches to divide data into homogeneous groups and make decisions based on specific clinical characteristics [26]. Authors’ recommended pathway for reading AI research These models are accessible through libraries like Scikit-learn (Python) and Caret (R) [27, 28]. Distance-based models like KNN tend to perform better with continuous data because they rely on measuring distances between data points. While KNN can be adapted for categorical data, it often requires additional preprocessing, such as encoding categories. In contrast, tree-based models, such as decision trees or random forests, handle both continuous and categorical data effectively. They are particularly well-suited for categorical variables, as they create splits that group similar values together, but they also perform well with continuous data by using thresholds to define splits [29]. Unlike classification models, regression models predict continuous numeric values [30] (Fig. 1). In anesthesiology, they can predict postoperative care needs, length of hospital stay, or optimal anesthetic doses [31, 32]. These models are based on statistical theories such as linear and polynomial regression, which use least squares methods and analysis of variance (ANOVA) [33]. These models can be accessed through libraries like Statsmodels and Scikit-learn (Python). ML models are constantly evolving, and new approaches to traditional models have been developed, such as ensemble models or optimized models, capable of detecting patterns in more sophisticated datasets [34]. When implementing a computational model to predict a clinical outcome, several models are tested. Rigorous ML model evaluation requires strategic data partitioning into training and testing sets. This separation enables performance assessment on unseen data, mitigating overfitting and yielding realistic estimates of real-world performance. Whether through internal dataset division or external testing sets, this methodological distinction proves fundamental for validating model generalizability through performance metrics [35]. Hyperparameter optimization represents another critical element in ML development. Unlike model parameters derived during training, hyperparameters require pre-training specification to govern the learning process. These external controls significantly impact model performance by mediating the balance between underfitting and overfitting. Reliance on default library settings or suboptimal hyperparameter selection compromises model generalizability and predictive capabilities [36]. The model offering the best performance based on metric evaluation is selected as the final model with the best approach for each paper. However, other evaluated models inside the same studies can still be useful in different clinical contexts or with specific refinements [37, 38]. Applications of machine learning in anesthesiology ML has been used in Anesthesiology in numerous studies that have employed computational models to address complex clinical problems [39–41]. The following are relevant examples of ML applications, we highlight specific models that have been used in each study. Anesthetic dosage optimization Miyaguchi et al., in their study, predicted the need to increase intraoperative remifentanil infusion rates. The study evaluated various models, including logistic regression, SVM, random forest, LightGBM, ANN, and LSTM, with the latter showing the best performance. SHAP values, a technique that quantifies individual feature contributions to model predictions, were applied to analyze key variable contributions. Results showed that systolic blood pressure was particularly relevant, contributing more to the models, on average, than mean arterial pressure. Additionally, weight, heart rate, and the immediately preceding remifentanil flow rate proved to be important factors, as expected. Contrary to previously accepted ideas, age was not a notable contributing factor [42]. Hypotension Hatib et al. used machine learning to map arterial pressure waveform characteristics to predict hypotensive events. They chose logistic regression as the primary model for its ability to provide an interpretable prediction and allow smooth transitions between two classes: presence or absence of a hypotensive event. This approach generated a numerical score reflecting the severity of hypotension. The “logit” transformation was applied to the binary dependent variable, and linear regression was performed, which allows for continuous risk calculation. The developed model captured a large set of features calculated from high-fidelity arterial pressure waveforms to predict an upcoming hypotensive event (mean arterial pressure < 65 mmHg) [43]. This algorithm is currently used in The AcumenTM Hypotension Prediction Index (HPI) software (Edwards Lifesciences; Irvine, CA, USA) [44]. Arterial and central venous catheterization Suh et al. developed a predictive model to guide preoperative decisions on the use of arterial or central venous catheters for intraoperative monitoring and management. This decision tends to be easy for experienced anesthesiologists, however it can be complex for recent graduates or those in training as discussed by the authors. To develop the model, data from 66,522 surgical patients was retrospectively analyzed, including demographics, preoperative laboratory tests, surgical procedure characteristics, and catheterization specifications, extracted from the hospital’s electronic medical record (EMR) system. Given the unstructured nature of the EMR, they employed a deep neural network for extraction and classification, achieving good performance in predicting the outcome of interest. SHAP values were used to evaluate the individual impact of variables on the model’s prediction. Among the predictive variables with the greatest contribution were general anesthesia with endotracheal intubation, ASA score, preoperative glucose, and preoperative platelets [45]. Postoperative nausea and vomiting Xie et al. worked on predicting postoperative nausea and vomiting (PONV), a common issue affecting recovery that can potentially lead to serious complications. Early prediction of PONV is crucial for implementing preventive measures and guiding management. Using structured data from 2,222 patients, including perioperative variables such as surgery type, age, sex, weight, and infusion volume, the study developed 54 predictive models using algorithms like logistic regression, random forest, SGD, XGBoost, and CatBoost. SHAP Values helped identify the most relevant variables and facilitate model interpretation in a clinical context, with varying findings on different models. These findings were subsequently published for use [46]. Acute kidney injury (AKI) In their study, Luo et al. focused on predicting AKI in pediatric patients after cardiac surgery. AKI after cardiac surgery has an incidence ranging from 5 to 42% and is associated with morbidity, mortality, and prolonged hospital stays. Early prediction of AKI is crucial for implementing preventive interventions that minimize kidney damage and improve outcomes. Using demographic variables, laboratory results, and intraoperative characteristics, the study created and evaluated ML models such as KNN, SVM, random forest, and XGBoost. Interpretability techniques, like SHAP VALUES, were applied to break down each variable’s contribution to the model’s predictions. The analysis revealed better performance in a model that included preoperative and intraoperative variables. Among the most influential variables were baseline serum creatinine, perfusion time, body length, surgical time, and intraoperative bleeding [47]. In a similar study, Nagy et al. addressed AKI prediction in pediatric patients after cardiac surgeries. Structured data from 402 patients was used and organized in a structured format, including preoperative, intraoperative, and immediate clinical data. The LightGBM model showed the best performance for predicting AKI on the second postoperative day (POD2), with SHAP Values identifying the most influential variables. The most significant predictors included preoperative serum creatinine, total surgery duration, serum pH on postoperative day 0, serum lactate on the same day, and cardiopulmonary bypass duration [48]. Evaluation metrics The evaluation of ML models is crucial to determine their performance and suitability in clinical contexts. Different metrics are used based on the type of model that is being evaluated. Metrics for classification models In classification models, which aim to assign categories or classes to data, the most commonly used evaluation metrics are precision, recall, specificity, accuracy, F1-Score, and the area under the ROC curve (AUC-ROC). These metrics are expressed in values ranging from 0 to 1, where proximity to 1 denotes better performance [49, 50]. (Table 1). Table 1Metrics used to compare classification modelsMetricDefinitionRangePrecisionProportion of true positives among predicted positives[0–1]RecallProportion of correctly identified true positives[0–1]SpecificityProportion of correctly identified true negatives[0–1]AccuracyProportion of all correct predictions[0–1]AUC-ROCAbility to distinguish between classes at different thresholds[0–1]F1-ScoreCombines precision and recall into a single metric[0–1] Metrics used to compare classification models Precision or Positive Predictive Value: The proportion of true positives among all cases classified as positive (see Eq. 1). Recall or Sensitivity: The proportion of true positives among all actual positive cases (see Eq. 2). Specificity: The proportion of true negatives among all actual negative cases (see Eq. 3). F1-Score: Combines precision and recall into a single metric, providing a balance between both. It is especially useful in contexts where balancing the importance of identifying true positives and minimizing false positives is desired (see Eq. 4). Accuracy: Measures the proportion of correct predictions, both positive and negative, out of total predictions made (see Eq. 5). AUC-ROC: Evaluates the model’s ability to distinguish between positive and negative classes at different decision thresholds. An AUC value above 0.7 indicates good discriminative power, while values of 0.8 or higher suggest excellent classification performance. An AUC-ROC value of 0.5 characterizes a null model generating random predictions without discriminative capability, establishing the baseline for predictive performance assessment [50]. While AUC-ROC provides a single performance measure, threshold-dependent metrics (precision, recall, specificity, and F1 score) fluctuate based on selected decision boundaries. Optimal threshold selection demands careful consideration of context-specific trade-offs. In critical medical scenarios, where missing true positives carries severe consequences, thresholds prioritizing high recall prove essential. Alternatively, contexts where false positives incur significant costs necessitate thresholds favoring precision. These inherent trade-offs preclude universal benchmark values for threshold-dependent metrics, requiring evaluation within the specific application context. In imbalanced datasets, where class distribution significantly differs, traditional performance metrics may mislead evaluation efforts. Although AUC-ROC incorporates both true positive and negative rates, it can inflate performance scores despite poor minority class prediction. The precision-recall (PR) curve offers more robust evaluation by focusing specifically on minority class performance, particularly valuable when rare outcomes hold clinical significance. Consequently, the area under the PR curve (AUC-PR) often provides more reliable performance assessment than AUC-ROC for imbalanced data [51]. Calibration curves serve as crucial performance indicators in medical applications by quantifying the alignment between predicted probabilities and observed outcomes. Proper calibration ensures predictive accuracy, a predicted 70% probability should correspond to a 70% occurrence rate. This alignment proves essential for clinical decision-making, where reliable probability estimates directly impact patient care [52]. Metrics for regression models For regression models, which predict continuous numerical values, the most common evaluation metrics are Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and R-squared (R²). These metrics are measured in values ranging from 0 to 1 as well. Unlike classification metrics, proximity to 0 denotes better performance, except for R² for which higher values indicate better performance [53]. (Table 2). Table 2Metrics used to compare regression modelsMetricDefinitionRangeMSE (Mean Squared Error)Average of squared errors[0- ∞+]RMSE (Root Mean Squared Error)Square root of MSE[0- ∞+]MAE (Mean Absolute Error)Average of absolute differences[0- ∞+]R-squared (R²)Degree of fit between predicted and actual values[0–1] Metrics used to compare regression models Mean Squared Error (MSE): Calculates the squared errors between the model’s predictions and the actual values. It helps understand how much the predictions deviate, penalizing large errors. Lower values indicate better performance, with an MSE close to 0 being ideal. Reference values can vary from small fractions to larger units depending on the specific data context [53]. Root Mean Squared Error (RMSE): Is the square root of MSE and is in the same units as the predicted data, allowing a more direct interpretation of the average error size. An RMSE close to 0 indicates good performance. The smallest possible value is sought, with the scale dependent on the specific problem and data magnitude [53]. Mean Absolute Error (MAE): Calculates the average of the absolute differences between predictions and actual values. It doesn’t penalize large errors more than small ones, providing a clear measure of average error. A low value indicates good fit, and values close to 0 are ideal. The MAE value can vary according to the data range, but generally should be as small as possible [53]. R-squared (R²): Indicates how well the predicted values fit the actual values. A value close to 1 suggests a good fit, indicating that the model explains much of the data variance. An R² of 0 indicates that the model explains none of the variability, while negative values can occur if the model is worse than a simple average baseline. In well-fitted models, R² is expected to be close to 0.8 or higher [53]. Importance of selecting appropriate metrics in the clinical context and interpretation Choosing appropriate metrics in ML is crucial to ensure models are useful and relevant in the clinical context. Classification metrics evaluate model performance and can reveal problems of overfitting or underfitting. Overfitting refers to a model with high accuracy on the training set but performing poorly on new data, with low recall and F1-Score metrics, indicating the model is not able to generalize. Conversely, underfitting refers to a model showing low accuracy and recall on both datasets, which can be due to model simplicity, not being able to capture complex patterns in the data, resulting in poor performance across all metrics [54]. In critical clinical applications, such as disease detection, recall is crucial for identifying all positive cases, but must be balanced with specificity to avoid a high number of false positives. Achieving this balance is essential to prevent overfitting or underfitting, ensuring the model’s robustness and applicability in clinical practice [55, 56]. In regression models, focusing on metrics like RMSE or MAE is necessary for understanding how prediction errors impact clinical decision-making. A low RMSE indicates that predictions are, on average, close to actual values, which can be essential in precise medication dosing. Additionally, metrics like R-squared (R²) can provide an overview of model fit but should be interpreted cautiously, especially in situations with high data variability [57, 58]. In the following section, we detail the performance results of each model previously discussed. This analysis reveals the strengths and limitations of each approach, providing a foundation for their theoretical integration into practice. However, these models currently exist only in developmental form, still needing extensive validation prior to clinical implementation. Other authors have thoroughly examined these validation prerequisites [59, 60]. Performance metrics of previously discussed models Miyaguchi et al. demonstrated that the LSTM model was most effective in predicting anesthetic infusion events, specifically related to increased remifentanil administration. The model achieved an AUC-ROC of 0.753, with a sensitivity of 0.659 and specificity of 0.732, reflecting its moderate ability to correctly discriminate between infusion events and non-events [42]. Hatib et al. developed a machine learning-based algorithm to predict arterial hypotension using 3,022 features per cardiac cycle. The model achieved 88% sensitivity and 87% specificity in predicting the outcome by 15 min, with an AUC of 0.95, indicating that the model correctly identifies most hypotension cases (high sensitivity) and also minimizes false positives (high specificity). In the 5 min mark prediction, these metrics increased to values close to 92%, with an AUC of 0.97, reflecting an even greater ability to accurately predict hypotensive events. Additionally, the algorithm was able to detect dynamic changes in physiological interactions, specifically between left ventricular contractility, preload, and afterload, improving the accuracy of hypotension prediction. These metrics indicate that the model is not only effective but also captures the underlying physiology of hypotensive events, making it a valuable tool for clinical practice [43]. Suh et al. employed a deep neural network to predict the need for arterial and central venous catheterization in non-cardiac surgeries. The model achieved an AUC-ROC of 0.92, indicating excellent ability to distinguish between classes. The AUPRC metric showed a value of 0.79, reflecting a good overall balance between precision and recall. However, the F1-Score of 0.43 suggests possible difficulties in discriminating false positives, indicating that the model might have problems correctly identifying all positive cases without generating too many errors in predictions [45]. Nagy et al., using the LightGBM model, showed promising performance in predicting AKI in pediatric patients after cardiac surgeries. In testing, the model achieved an accuracy of 0.91 and an AUC-ROC of 0.88, reflecting high discriminative ability. However, the recall was 0.63, suggesting it may still miss some positive cases. The F1-score of 0.73 indicates a good balance between precision and sensitivity, although the lower sensitivity reflects the need for improvements in identifying positive cases [48]. In Luo et al.‘s study, the XGBoost model proved most effective in predicting acute kidney injury (AKI) in pediatric patients. The model achieved an AUC-ROC of 0.890 in the derivation cohort and 0.857 in external validation using preoperative variables, improving to 0.912 and 0.889 respectively when including intraoperative variables, reflecting excellent discriminative ability. With a low-risk cut-off value, the model obtained 95% sensitivity and 98.3% predictive value, minimizing false negatives. For high-risk patients, it achieved 95% specificity, demonstrating its effectiveness in identifying AKI cases [47]. Xie et al., using the CatBoost model, achieved the best performance in predicting postoperative nausea and vomiting (PONV) in patients using patient-controlled analgesia. The model achieved an AUC-ROC of 0.95, accuracy of 0.86, precision of 0.89, and recall of 0.81. Additionally, the F1-score of 0.85 and specificity of 0.90 demonstrated a solid balance between accurately identifying patients with PONV and minimizing false positives. These results highlight the model’s potential to improve early prediction of postoperative complications [46]. Explainability in machine learning In clinical practice, the interpretability of ML models is essential to ensure that medical decisions based on these models are reliable and comprehensible. Unlike other fields, in medicine, each decision has a direct impact on patient health. For this reason, healthcare professionals need to understand the predictions made by the models and also the underlying rationale behind these predictions [61]. While interpreting performance metrics is fundamental to evaluating a model’s performance, this alone is not sufficient to achieve complete interpretability. Metrics provide a quantitative view of the model but do not explain how or why the model makes specific decisions. It is crucial that healthcare professionals can understand the specific factors influencing the model’s predictions, which is vital for justifying and validating clinical decisions [62]. Some ML models offer inherent interpretability. Logistic and linear regression models reveal feature contributions through individual coefficients [63], while decision trees provide intuitive visual hierarchies of predictive decisions [64]. However, increasing model complexity, particularly in ensemble methods and deep learning architectures, diminishes interpretability and obscures individual prediction pathways. Such complexity creates a need for supplementary analytical tools to elucidate model mechanics and maintain transparent, justifiable predictions for clinical applications. Interpretable models allow physicians to justify their decisions based on clear and understandable criteria, which is especially important in critical situations where accuracy is crucial. Additionally, interpretability facilitates the identification of potential errors or biases in the model, allowing for adjustments before incorrect decisions can negatively affect patients. Thus, although interpreting performance metrics is important, a deeper and more detailed understanding of the model’s internal functioning is required to ensure that decisions derived from ML are safe, justified, and accepted by healthcare professionals [65]. SHAP values (Shapley Additive explanations) SHAP values are a tool in the ML field that significantly contribute to model explainability, especially for those traditionally considered “black boxes” such as neural networks and ensemble models. Based on Shapley values from game theory, SHAP values assign a fair contribution to each feature in a model, explaining the prediction process as a cooperative game between variables. These values can be calculated at two distinct levels: the population level, representing aggregate feature importance across the entire dataset, or the individual level, determining specific feature contributions for each case. In ML, this allows decomposing a model’s prediction into individual contributions of single features from the dataset, indicating how each can positively or negatively influence the final outcome. This approach can simplify the interpretation of complex models and provide a clear, quantifiable explanation of decisions inside a model [66]. For example, in a model predicting the probability of postoperative complications, SHAP Values can show how much factors like patient age, hemoglobin value, or specific test results contribute to the final prediction [67]. This allows healthcare professionals to better understand the reasons behind a prediction and make more informed and transparent decisions [68]. The ability of SHAP Values to offer detailed and accessible explanations makes ML models more reliable and comprehensible, which is essential in a clinical environment where decisions must be justified and understood by all involved in patient care. Thus, SHAP Values not only improve model interpretability but also promote the adoption and trust in ML tools in medicine [69]. In section “Performance Metrics of Previously Discussed Models”, the article’s objective was explained [45], along with the model used to predict the need for arterial and central venous catheterization in non-cardiac surgeries. Section 3 addressed the model’s key metrics, highlighting its performance in terms of AUC-ROC, AUPRC, and F1-Score. Below, the SHAP Values obtained in the study will be presented, offering a detailed explanation of each feature’s influence on the model’s predictions, improving the understanding and interpretability of the results in the clinical context. The results of the analysis using SHAP values show the most important features that influenced the deep neural network (DNN) model in predicting the need for invasive catheter insertion. In predicting the need for an arterial catheter (Fig. 2A), the most important feature was ASA-PS class III, indicating that patients with higher anesthetic risk, according to this classification, are more likely to require an arterial catheter. In contrast, for predicting central venous catheter insertion (Fig. 2B), the most relevant factor was the operation code for pylorus-preserving pancreatoduodenectomy (PPPD), highlighting the importance of the type of surgical procedure in the decision [45]. DNN SHAP values [45] These graphs helped visualize the model’s decision sequence and the key factors that influenced the final prediction, which increased the model’s transparency and facilitated its interpretation in the clinical context. Figure 1 shows the authors’ recommended pathway for reading AI Research (Fig. 1). Conclusions Critical evaluation of ML articles in anesthesiology is essential to ensure these tools can be used effectively and safely. Performance metrics provide information on the accuracy, sensitivity, specificity, and discriminative ability of models, allowing interpretation of how they balance correct identification of positive and negative cases. The interplay between these metrics helps evaluate the overall performance of models and their clinical applicability. Moreover, explainability which can be understood by tools like SHAP Values, is vital for understanding how each feature influences predictions, facilitating justification of and trust in decisions. It is also crucial to consider ethical aspects related to the use of AI in anesthesiology, focusing on ensuring that artificial intelligence algorithms are transparent in their processes and results, mitigating their “black box” nature, allowing users to understand and validate their decisions. Transparency in model development and interpretation is crucial for maintaining trust in technological tools and ensuring that their implementation respects ethical principles and benefits the patient without compromising their safety or well-being. Acknowledgements Not applicable. Abbreviations Artificial Intelligence Deep Learning K-Nearest Neighbors Machine Learning Shapley Additive Explanations Supporting Vector Machine Author contributions FOO interpreted and wrote sections regarding AI models and metrics, participated in revision process. SA-R interpreted and wrote sections regarding medical applications and was responsible for figure creation, participated in revision process. TEMV participated in writing the conclusions and assisted in writing sections on AI models and metrics. GAC-S provided general oversight of the project and refined the writing throughout all sections of the manuscript. All authors participated in the literature search. All authors read and approved the final version of this manuscript. Funding This research did not receive any funding from agencies in the public, commercial, or not-for-profit sectors. Data availability No datasets were generated or analysed during the current study. Declarations Ethics approval and consent to participate Not applicable. Consent for publication Not applicable. Competing interests The authors declare no competing interests. Clinical trial number Not applicable. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement No datasets were generated or analysed during the current study.