Sustaining High Reliability Amid Artificial Intelligence Adoption in Oncology Authors: Shawn Stapleton, Renu Sara Nargund, Caroline Chung Published: 2026 Introduction Artificial intelligence (AI) is evolving from mechanistic and statistical systems designed for narrow, well-defined problems to potentially adaptive, agentic architectures with the prospect of perceiving, reasoning, interacting, and generating information.1 These technologies are increasingly embedded across health care with the promise of improving decision making, workflow efficiency, and operational performance; however, the introduction of such technologies into the workflow presents new risks.2 Reliability in the quality and safety of clinical practice in an AI-enabled environment is becoming increasingly complex, requiring effective human oversight and workflow integration, moving beyond purely technical validation.3 Clinical teams are now expected to manage usage, safety, and impact of AI tools, often termed the human-in-the-loop, yet they are not properly enabled with adequate training or support. Oncology represents a microcosm of the challenges that define clinical AI reliability and precision. Cancer care involves longitudinal tracking along the cancer journey, clinical decision making amid narrow therapeutic margins, and frequent transitions across inpatient, outpatient, and procedural settings. As a result, AI systems need to consistently deliver outputs across the cancer care continuum, spanning from screening, diagnosis and staging, treatment planning, treatment delivery, and longitudinal inpatient, emergency, and ambulatory care. Empirical evidence demonstrates that AI systems deployed in oncology can exhibit reliability failures that are not readily apparent to end users. Models validated in general medical populations may silently underperform in cancer patient cohorts, but outputs are often presented to clinicians without visible indications of the difference in performance. As an example, generative AI–based ambient scribes are now among the most rapidly adopted AI applications across health care.4 The unique oncology-specific demands and complexities of capturing and meaningfully integrating the relevant and critical clinical documentation to inform high-stake decisions, particularly with clear coordination across several provider-patient interactions, are challenging and introduce risks. AI-generated documentation errors, including omissions, misattributions, or hallucinations, may appear plausible to the reviewing clinician. Once finalized, they are embedded into a longitudinal record where they can shape downstream decisions without any indication of their AI origin.5,6 Similarly, AI systems embedded in cancer-specific workflows, such as radiation therapy auto-segmentation or assisted diagnostics in radiology or pathology, can generate plausible but incorrect outputs that may be accepted during routine use possibly because of automation bias.7 Together, these findings indicate that reliability failures in oncology remain vulnerable at the human-AI interface, where the capacity to detect and intercept AI outputs determines whether AI enhances or undermines cancer care. Addressing the unique reliability challenges introduced by AI requires continuous detection and mitigation. For instance, models drift as the data, workflows, clinical practices, populations, and diseases evolve, raising risks of errors and harms that may go unnoticed without systematic surveillance. Concurrently, the cognitive overload from growing digital responsibilities and interfaces with the advent of digital medicine can be further exacerbated by increasing responsibilities assumed by clinical care teams acting as human-in-the-loop validators of AI outputs in real time. Oncology care delivery teams may overtrust or reject AI outputs based on workload, opinions, or anecdotal experiences, further complicating the integration of AI into health care.8 Evidence from prior technology deployments, particularly the electronic health record, in health care shows that harms can emerge and persist after deployment.9 Human-in-the-loop management of AI may fail silently without proper frameworks in place. These breakdowns rarely stem from a single-point defect; they emerge from complex interactions that propagate across people, processes, and technologies.10 Ensuring reliability of AI solutions, therefore, is not a one-time engineering challenge but a continuous sociotechnical function at the human-AI interface. Accordingly, we propose a framework that integrates high-reliability organization (HRO) principles with the Swiss Cheese model to operationalize the capable human-in-the-loop AI to help ensure the safe and impactful use of AI in clinical care. AI as a Reliability Challenge AI introduces distinct reliability challenges in health care because it relies on consistently generated data, is often built to integrate complex information without full context, and relies on the person receiving the model output to consider all the uncertainties in the decision-making process. The reliability challenge has been compounded with emerging use of generative AI models, which can potentially adapt and evolve with exposure to new data and contexts, such that their behavior is not fixed. Models can degrade silently, shift performance boundaries, or interact with human and system factors in unexpected ways.11 Reliability is inherently a moving target dependent on the data, context, and ongoing human interpretation and oversight. Vulnerabilities in reliability arise at multiple levels. At the data layer, variability or changes in data quality, introduction of bias, and shifts in patient populations, documentation practices, or data sources can alter data distributions and result in data drifts that cause AI outputs to deviate from expectations, as shown in sepsis and COVID-19 predictive models.12-14 Ambient scribes have generated lower-quality summaries when patients switch languages mid-consultation and discuss multiple complaints or when consultations involve follow-up rather than new visits.15 Each of these instances represent a data input shift that degrades model output without any visible warning to the clinician. At the algorithmic layer, models performing well in development may fail under real-world contexts, encountering edge cases, hallucinations, or adversarial inputs.16 In oncology, such failures carry particularly high stakes. In one illustrative scenario, an ambient scribe incorrectly documented that a patient with metastatic non–small cell lung cancer had failed both first- and second-line therapies when only first-line therapy had been completed; a subsequent oncologist, relying on these notes, initiated third-line immunotherapy prematurely, resulting in severe immune-related pneumonitis and a worsened prognosis.17 These risks underscore the need for local validation, potential fine-tuning, and continued monitoring.18 At the human layer, automation bias, over-reliance, alert fatigue, and probabilistic misinterpretation can suppress vigilance and impair detection of abnormal AI behavior. Prolonged automation may erode or prevent skill development, diminishing situational awareness and confidence in manual reasoning.18,19 A recent study on AI-assisted polyp detection demonstrated measurable deskilling.20 Such findings show that automation can reshape attention, skills, and decision-making behaviors, reinforcing the need for deliberate preservation of clinical competence.21-23 Beyond risks at each layer, variable interactions across layers can result in complex, compounding risks. A small data drift can trigger algorithmic instability that goes unnoticed because of cognitive overload or misplaced trust, allowing errors to propagate through workflows.24,25 Reliability cannot be ensured through technical validation alone, especially in an era of evolving US Food and Drug Administration regulatory oversight. It depends on a capable human-in-the-loop who recognizes weak signals, interprets uncertainty, and intervenes when AI behavior deviates from expectation. Equally important, institutional defense barriers must provide dynamic safeguards that catch what individual clinicians cannot through pathways that evolve as AI and workflow conditions change. The Capable Human-in-the-Loop The capable human-in-the-loop represents the human interface to clinical care delivery that also serves as a core adaptive layer of defense in an increasingly automated health care system. AI reliability therefore depends on clinical teams evolving from technology users to active coregulators who are capable of interpreting, contextualizing, and intervening when AI behavior deviates from expectations.26,27 Ambient clinical documentation illustrates the critical distinction between a nominal and a capable human-in-the-loop. Clinicians are required to review and attest to AI-generated notes before finalizing them, yet automation bias increases with familiarity, and high caseloads create pressure to approve drafts with minimal scrutiny. A capable human-in-the-loop, by contrast, maintains the critical lens along with clinical reasoning and verification skills needed to detect and correct for even subtle inaccuracies amid the clinical time pressures. Reason's28 Swiss Cheese model illustrates the evolving role of clinical teams in managing AI reliability. Each layer of defense in health care, including regulatory, organizational, technological, and human, contains vulnerabilities through which hazards can pass when misaligned.29 AI now occupies a dynamic component of the technological layer, which can both serve as a defense that enhances performance and as a source of risk to reliability, quality, and safety (Fig 1). AI systems introduce dynamic holes across both the human and AI layers, where holes may appear, move, grow, or shrink, reflecting amplified, emerging, stochastic, or shifting vulnerabilities that can dynamically align at the human-AI interface. These holes correspond to real-world AI reliability gaps such as bias, data drift, and stochastic error, potentially then coupled and compounded by human factors like trust calibration and over-reliance that can evolve over time.30 Enabling a capable human-in-the-loop: Applying high-reliability principles to AI safety. Adaptation of Reason's Swiss cheese model applied to the human-AI interface in clinical care. Harm propagates from left to right through regulatory, organizational, and operational layers, shown in light yellow as upstream layers of defense. AI is depicted in blue as a distinct layer although it sits within the broader technology environment and together with the capable human-in-the-loop and defense barriers forms the human-AI interface. The combination of AI and human vulnerabilities, shown in the lower panel, requires a dynamic defense enabled by HRO principles: Anticipate, Observe, Verify, Recover, and Escalate. The capable human-in-the-loop functions as an integral safety layer equipped with HRO principles can intercept harm before it propagates further. Defense barriers embody HRO principles to intercept additional vulnerabilities as they emerge and shift amid the evolution of the human and AI interaction. AI, artificial intelligence; HRO, high-reliability organization. In this new ecosystem, the distinction between being present in the loop and being reliably protective in the loop becomes critical. A human-in-the-loop is anyone who interacts with AI output; a capable human-in-the-loop has AI and data literacy to monitor for performance drift,31 interpret probabilistic outputs accurately,32 calibrate trust appropriately, and know when to pause, override, or question AI predictions.10 These competencies can be strengthened without explicit AI modeling or coding expertise through guidelines, structured feedback, and deliberate exposure to AI failure scenarios.33 Beyond sustaining these competencies, dynamic defense barriers include processes and HRO practices that catch vulnerabilities as they accumulate and redistribute oversight responsibility beyond the individual clinician. The HRO framework provides a structured framework to institutionalize sustainable vigilance, adaptability, and learning by embedding human reliability functions into routine operations (Fig 1).34 High Reliability in the Era of AI The HRO principles are a foundation for managing complexity and uncertainty in safety-critical environments.34 They focus on detecting weak signals, sustaining vigilance, and maintaining safe performance under changing conditions.35 HRO naturally aligns with Reason's Swiss Cheese model,28 which conceptualizes safety as layered defenses that can fail when vulnerabilities align across interfaces, including the human-AI interface. This transforms reliability from a static engineering property into an adaptive operational framework that enables the capable human-in-the-loop. The alignment between each HRO principle, corresponding AI reliability challenges, and the human capabilities is summarized in Table 1, providing a practical guide for translating HRO theory into institutional practice (Fig 1). TABLE 1.HRO Principles to Ensure AI Reliability in Health Care by Enabling a Capable Human-in-the-LoopHRO PrincipleGoalAI Reliability ChallengesRelevant High Reliability SkillsetsClinical ExamplesPreoccupation with failureAnticipate, detect, and address weak signalsSilent performance drift without alerts. Hidden uncertainty obscuring abnormal outputs. AI-clinician discrepancies overlooked under workloadMonitor weak signals of degradation, including drift and overridesEscalate persistent anomalies in AI outcomes or usage patternsEstablish channels to strengthen weak signals once identifiedA clinician highlights rising edit rates on AI-generated notes or recurring omission of examination findings signaling degradation. Risk management solutions are activated to prevent error propagationReluctance to simplifyAvoid treating probabilistic outputs as deterministic truthAutomation bias and overconfidence in outputs. Hidden data quality errors distort inference. Misinterpretation of probabilistic outputs as deterministic truthInterrogate AI outputs against context and domain logicChallenge assumptions, boundaries, and failure modesExplicitly communicate uncertainty and calibrate trust through routine challenges of AIA clinician notices a discrepancy between listed medications, clinical presentation, and an ambient scribe's reference to failed second-line therapy. Acting on the uncertainty, they review the source transcript, confirm that the therapy was only discussed, and edit the documentationSensitivity to operationsMaintain awareness of workflow, workload, and context shiftsHuman oversight outpaced by AI output volume. Misalignment between timing of AI outputs and clinical workflow. Cognitive overload increases automation relianceAdjust reliance on AI based on workload and operational conditionsEmbed AI status awareness into handovers and huddlesSurface workflow mismatches through nonpunitive reporting channelsWhen faced with a high patient volume, a clinician who regularly uses ambient scribing recognizes that additional time may be needed to carefully review notes and ensure accurate documentation. Unsigned note backlogs inform scheduling decisions to adjust time between appointmentsCommitment to resilienceSustain adaptive capacity when humans or AI failsOver-reliance leads to deskilling or never-skilling. Rapid error propagation outpaces human correction. Trust collapses after failureMaintain manual and cognitive competence independent of automationRapidly disengage or suspend unreliable AILearn from near misses through structured debriefs and recovery drillsAn auto-contouring tool is found to have generated results that are not accurate. Rather than having trust collapse, a deeper dive into the root cause of the failure can help explain the results and identify ways to alleviate the problem, both strengthening the value of human oversight and re-establishing trust in the AIDeference to expertiseShift authority to the most reliable source in contextUnclear authority when AI and clinicians disagree. Opaque reasoning impairs trust calibrationNormalize challenge to AI recommendations regardless of hierarchyDefine clear escalation and override thresholdsDynamically reassign authority as human or AI reliability shiftsAn AI-generated radiology read misses a lesion previously flagged as indeterminate by a radiologist. The primary clinician flags the discrepancy, defers to the radiologist's prior interpretation, and ensures follow-up imaging proceeds. Institutional guidance on source hierarchy supports the override, and flagged discrepancies inform model performance reviewNOTE. The table aligns established HRO principles with AI-specific reliability challenges and the human capabilities needed to anticipate, detect, and recover from AI system vulnerabilities.Abbreviations: AI, artificial intelligence; HRO, high-reliability organization. HRO Principles to Ensure AI Reliability in Health Care by Enabling a Capable Human-in-the-Loop NOTE. The table aligns established HRO principles with AI-specific reliability challenges and the human capabilities needed to anticipate, detect, and recover from AI system vulnerabilities. Abbreviations: AI, artificial intelligence; HRO, high-reliability organization. Preoccupation with failure assumes that harm is foreseeable unless actively mitigated. Silent degradation such as data or algorithm drift, cognitive overload, or integration errors can erode reliability without warning. Clinical teams must maintain awareness of weak signals such as implausible outputs, increasing overrides, or alerts that the AI is operating out of bounds. These signals should trigger verification, escalation, or fallback. Training clinical teams to identify and act on early indicators ensures vigilance functions as a dynamic safety layer that prevents harm. Reluctance to simplify ensures reliability through deliberate verification of AI outputs against patient context and domain logic to prevent the indiscriminate acceptance of AI outputs as deterministic truth. Under cognitive load, clinicians may default to automation or confirmation bias, overconfidence, and accepting AI recommendations that appear certain. Structured communication of disagreement, through the Situation-Background-Assessment-Recommendation framework, reinforces divergence between clinical and AI reasoning as a signal for review, not inefficiency.36 To further enable human-in-the-loop, interfaces should display predication uncertainty, data quality input warnings, and validation prompts that support trust calibration. Sensitivity to operations emphasizes continuous awareness of how workflow, workload, and context influence AI reliability. Human and AI performance fluctuates with documentation timing, staffing levels, and environmental stressors. Cognitive overload can increase reliance when AI reliability is silently deteriorating or through workflow misalignment, such as AI outputs appearing too early or too late for safe use. Integrating AI status checks into huddles and escalating usability mismatches through Just Culture channels allow early human recalibration before small misalignments evolve into systemic failures.37 Commitment to resilience ensures that both humans and AI can recover when either one fails. Over-reliance breeds deskilling or never-skilling, weakening fallback capabilities. Resilience requires regular AI-off drills, failure mode refreshers, and deliberate manual workflow practice to competency. Clinicians must have authority to override, suspend, or escalate AI concerns without penalty. Structured debriefings convert AI anomalies and harms through root cause analysis, reinforcing the ability to shift fluidly between automated and manual reasoning. Deference to expertise ensures that authority shifts to the most reliable source in context, whether human or AI. Safe performance depends on knowing when to trust AI and when human judgment should prevail. Reliability requires clear escalation pathways, defined overrise triggers, and explainability cues that reveal model confidence and rationale, aiding clinicians to calibrate trust dynamically based on situational reliability rather than appearance of AI certainty. Collaborative reasoning ensures that authority resides with the most reliable judgment at any moment. In conclusion, reliability in AI-enabled health care, particularly in the complex coordination of multidisciplinary care delivery in oncology, arises from the continuous, adaptive interaction at the human-AI interface. Application of HRO principles provides a foundation that enables health care organizations to sustain trust and safety with using frameworks that are already familiar and in place in most cancer centers and hospitals. Reason's Swiss Cheese model helps organizations operationalize these principles by enabling humans to map out, recognize, and mitigate vulnerabilities across the human-AI interface, preventing their convergence into emerging, amplified, and stochastic harms before they result in harm or failure.