Analysis of the challenges in ensuring robust and effective clinical evaluation of AI-based digital medical devices Authors: Sandrine Boulet, Vincent Thévenet, Judith Abecassis, Corinne Collignon, Bruno Giraudeau, Edouard Lhomme, François Petit, Raphaël Porcher, Laura Richert, Florence Saillour, Moreno Ursino, Gaël Varoquaux, Vincent Vercamer, Sarah Zohar, Rodolphe Thiébaut Published: 2026 ABSTRACT The rapid evolution of algorithms over time, the validity of the digital criteria used for evaluation and the organisational impact are examples of issues that are both diverse and more specific to digital medical devices (DMDs) than drugs. Furthermore, some DMDs that use artificial intelligence (AI) have the feature of being developed using continuous learning systems, in which the algorithms learn from all the data they use. Despite these unique features, their clinical evaluation often follows existing frameworks designed for drugs. Nevertheless, the specific features of AI-based DMDs call for an adapted clinical evaluation approach and raise new methodological questions.We propose a narrative review of these characteristics and the challenges they could imply for the evaluation of DMDs.Hence, the definition of users and intended use, the choice of datasets for analytical and clinical validation, the specification of outcomes of interest, the use of real-world data studies in addition to randomised clinical trials, and the choice of an appropriate comparator are key elements that can influence the design, analysis and generalisability of the clinical assessment. WHAT IS ALREADY KNOWN ON THIS TOPIC While digital medical devices (DMDs) are often clinically evaluated in a framework similar to that of drugs, they have many specific features of their own, which means that their entire evaluation process needs to be rethought differently from that of drugs and for each subtype of DMDs according to intended use and user. While digital medical devices (DMDs) are often clinically evaluated in a framework similar to that of drugs, they have many specific features of their own, which means that their entire evaluation process needs to be rethought differently from that of drugs and for each subtype of DMDs according to intended use and user. WHAT THIS STUDY ADDS This study summarises the specific characteristics of artificial intelligence (AI)-based DMDs compared with drugs and the methodological issues including the choice of endpoints and comparators that will influence their clinical evaluation design, analysis and the generalisability of results over time. This study summarises the specific characteristics of artificial intelligence (AI)-based DMDs compared with drugs and the methodological issues including the choice of endpoints and comparators that will influence their clinical evaluation design, analysis and the generalisability of results over time. HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY This study can be taken as a starting point to tackle the design choices and methodological resolution of the issues raised and thus propose a rigorous framework for the clinical evaluation of AI-based DMDs. This study can be taken as a starting point to tackle the design choices and methodological resolution of the issues raised and thus propose a rigorous framework for the clinical evaluation of AI-based DMDs. Introduction Digital medical devices (DMDs) are designed to prevent, diagnose, monitor, treat or alleviate disease effects. This device class includes smartphone apps, stand-alone software, online tools and algorithms that use data from medical devices such as medical imaging devices, sensors or monitors. Examples of DMDs include clinical decision support systems for healthcare professionals, physical devices paired with software such as closed-loop insulin pumps1 and apps for monitoring chronic diseases. Digital therapeutics (DTx), defined by the Digital Therapeutic Alliance as ‘evidence-based therapeutic interventions driven by high-quality software programmes to prevent, manage or treat a medical disorder or disease’, belong to the medical device software class. The use of DMDs—or software as medical devices (SaMDs)—is increasing2 due to opportunities generated by the digitalisation of medical data (eg, electronic health records, registries), technological innovations such as the Internet of Things (eg, smartwatches including health tech apps) and rapidly evolving artificial intelligence (AI) methodologies. During development, a DMD may seem promising based on robustness and resilience evaluated by analytical validation. According to the SaMD Working Group from the International Medical Device Regulators Forum (2017),3 analytical validation answers ‘Does your SaMD correctly process input data to generate accurate, reliable and precise output data?’, whereas clinical validation answers ‘Does use of your SaMD’s accurate, reliable and precise output data achieve your intended purpose in your target population in the context of clinical care?’ Analytical validation involves retrospective data analysis to evaluate whether the DMD produces accurate, robust and reliable results. However, only clinical evaluation and validation through interventional and observational studies can provide evidence to assess the risk–benefit ratio for individuals, healthcare professionals or healthcare systems.4 For CE-marking, clinical validation is mandatory for medium-risk to high-risk DMDs based on the medical device’s classification, where incorrect recommendations could impact prognosis (ranked IIb and higher) (https://webgate.ec.europa.eu/udi-helpdesk/en/other-relevant-information/medical-device-classification.html). Most clinical trials, study designs and methodologies were developed within a drug (ie, medicinal products) evaluation framework and are thus tailored to non-evolutive, non-complex health technologies.5 However, the fundamental difference when a drug is compared with a DMD is that DMD use is a complex intervention that can impact the care system beyond the patient or evolve over time, potentially altering its outcomes based on different versions released for users. Unlike drugs for which the final user is the patient, a DMD can have several users, including patients, healthcare professionals, nonhealthcare professionals (carers, family, nonmedical staff of a service provider, etc) or healthcare systems (hospitals, etc). Thus, the clinical evaluation paradigm must evolve and requires adapting methodological and statistical approaches for the specific DMD context. In recent years, guidelines and initiatives from academic consortia and health technology assessment regulators (HTAs) have provided recommendations for the clinical evaluation of DMDs.5 6 While there is significant literature on the analytical validation of DMDs based solely on retrospective data use,7 clinical validation is described only using overall good-practice guidelines without detailing clinical study endpoints, conduct and design.5 This paper aims to identify and enumerate key outcomes of interest and provide an in-depth discussion on designs for the comprehensive evaluation of AI-based DMDs, offering more detailed insights than current recommendations do. We propose highlighting technological, medical and feasibility challenges when evaluating and clinically validating these types of DMDs. Our approach details how stakeholders and the AI-based DMD ecosystem should be cautious when planning real-life clinical evaluations and validations and what aspects should be carefully considered. This paper is organised as follows. We first distinguish the study population for analytical and clinical validation requirements. Next, we define the endpoints. We then present challenges regarding designs and statistical methods. Subsequently, we address the follow-up question of different AI-based DMD versions over time. A discussion closes the manuscript. Study population For analytical validation, retrospective data from individuals meeting specific eligibility criteria (eg, diagnosis, age range)8 are typically partitioned into three subsets for algorithm training, validation and testing. In contrast, clinical validation relies on prospective data collected through clinical trials, in which participants are selected based on predefined inclusion and exclusion criteria tailored to the study objectives. Therefore, while both validation approaches involve data from individuals selected through eligibility criteria, the data sample characteristics often differ (see figure 1). When training, testing and validating the algorithm embedded in the AI-based DMD, developers (companies, academics, etc) need access to retrospective patient data, which must be sufficiently large and representative to obtain reliable, robust and accurate predictions. However, this can be challenging when considering (1) regulatory and data privacy policies, (2) data accessibility for analytical validation regarding the desired population characteristics linked to clinical associations9 and intended use of AI-based DMD and (3) feasible real-life clinical validation regarding clinical associations, intended use and users (see table 1). In fact, (1) accessing and using patient data for AI model development must comply with strict regulatory frameworks, such as the General Data Protection Regulation in Europe or the Health Insurance Portability and Accountability Act in the USA. These regulations impose stringent requirements on data anonymisation, data minimisation, patient consent and cross-border data sharing, potentially limiting the availability of diverse and representative datasets. Additionally, obtaining regulatory approval for AI in medical applications requires rigorous documentation and justification of data sources, which can further complicate the validation process. (2) The datasets used for training and validating AI models must accurately reflect the target population to ensure that the algorithm performs reliably across different demographic and clinical profiles. In addition to limited access to such data, due to fragmentation across healthcare institutions, biases in available datasets or the lack of sufficient cases for rare diseases exist. If the data used for analytical validation do not match real-world clinical conditions in which the AI-based DMD will be deployed, the risks of bias and poor generalisability increase, potentially affecting patient safety and device effectiveness. Finally, (3) beyond analytical validation, clinical validation requires testing the AI-based DMD by including the viewpoints of various stakeholders and considering, in real-world settings, the dynamic interactions between intervention and context, where interoperability and usability depend not only on algorithmic performance but also on how the system integrates into diverse clinical workflows, healthcare infrastructures, user interfaces and user confidence in AI-based tools. Illustration of the analytical validation, clinical evaluation and extrapolation between clinical evaluations for DMDs. Prior to the first clinical evaluation, analytical validation is performed. For analytical validation, retrospective data from individuals meeting specific eligibility criteria are typically partitioned into three subsets for algorithm training, validation and testing. In contrast, clinical validation relies on prospective data collected through clinical trials, in which participants are selected based on predefined inclusion and exclusion criteria tailored to the study objectives. Therefore, while both validation approaches involve data from individuals selected through eligibility criteria, the data sample characteristics often differ. Over time, the DMD is regularly updated. If extrapolations of clinical results between two versions of the device can be envisaged, the question arises as to when (and with what data) to carry out a new clinical evaluation. DMD, digital medical device; RWD, real-world data. Table 1Summary of the key challenges related to study population in the evaluation of AI-based DMDsKey challengesSummaryRegulatory and data privacy policiesRegulatory frameworks, such as the GDPR in Europe or the HIPAA in the USA, impose stringent requirements on:Data anonymisationData minimisationPatient consentCross-border data sharingDocumentation and justification of data sources Data accessibility for analytical validation regarding the desired population characteristics linked to clinical associations10 and intended use of AI-based DMD Limited access to such data (fragmentation across healthcare institutions, lack of sufficient cases for rare diseases…)Risks of bias and poor generalisability, potentially affecting patient safety and device effectiveness Feasible real-life clinical validation regarding clinical associations, intended use and users Inclusion of the viewpoints of various stakeholdersConsideration, in real-world settings, of the dynamic interactions between intervention and context, where interoperability and usability depend not only on algorithmic performance but also on how the system integrates into diverse clinical workflows, healthcare infrastructures, user interfaces, and user confidence in AI-based tools AI, artificial intelligence; DMD, digital medical device; GDPR, General Data Protection Regulation; HIPAA, Health Insurance Portability and Accountability Act. Summary of the key challenges related to study population in the evaluation of AI-based DMDs Data anonymisation Data anonymisation Data minimisation Data minimisation Patient consent Patient consent Cross-border data sharing Cross-border data sharing Documentation and justification of data sources Documentation and justification of data sources Limited access to such data (fragmentation across healthcare institutions, lack of sufficient cases for rare diseases…) Limited access to such data (fragmentation across healthcare institutions, lack of sufficient cases for rare diseases…) Risks of bias and poor generalisability, potentially affecting patient safety and device effectiveness Risks of bias and poor generalisability, potentially affecting patient safety and device effectiveness Inclusion of the viewpoints of various stakeholders Inclusion of the viewpoints of various stakeholders Consideration, in real-world settings, of the dynamic interactions between intervention and context, where interoperability and usability depend not only on algorithmic performance but also on how the system integrates into diverse clinical workflows, healthcare infrastructures, user interfaces, and user confidence in AI-based tools Consideration, in real-world settings, of the dynamic interactions between intervention and context, where interoperability and usability depend not only on algorithmic performance but also on how the system integrates into diverse clinical workflows, healthcare infrastructures, user interfaces, and user confidence in AI-based tools AI, artificial intelligence; DMD, digital medical device; GDPR, General Data Protection Regulation; HIPAA, Health Insurance Portability and Accountability Act. Endpoints As for drugs, clinical trial or study design theory requires specifying a major endpoint associated with the ‘clinical question’ and secondary endpoints providing supportive information about the effect on the primary endpoint. For medical device evaluation, it includes both clinical outcomes (efficacy—EFF, safety—SAF) and nonclinical outcomes (economic—ECO, organisational—ORG, social—SOC),10 even when the user is a healthcare professional or healthcare system. These endpoints can be gathered from various sources and tools, either as part of routine care or through digitally enabled add-ons as digital biomarkers: for example, utilising telemonitoring apps to gather patient-reported outcome measures or patient-reported experience measures.11 A DMD can collect a wide range of data directly. Some of these data can be used as outcome measures if the measurement is reliable and the evaluation criterion is relevant. This also offers the opportunity to provide real-time feedback, allowing postmarket surveillance.12 The algorithm embedded in the DMD may also lead to an additional layer of complexity. For example, reinforcement learning-based decision systems require retrospective datasets containing enough optimal or near-optimal decisions. However, suboptimal decisions can be difficult to identify, especially when clinical outcomes are distal from the decision point, thus reducing the quality of the training signal. Some endpoints are more specific to DMDs and complex interventions than to drugs, such as organisational impact (ORG).13 According to Roussel et al,14 organisational impact assessment examines the consequences (upstream and downstream) of introducing a medical device in terms of resources, production processes, delivery and information/training from a given perspective. Taking this into account for a DMD may involve, for example, being able to process signals sent by a user to a medical centre or doctor at any time. It may be necessary to manage the alert. This also requires fast human-machine interactions and connections. If there is a problem in this process—eg, the patient receives no response—they may think everything is fine. This type of problem relates to process studies conducted in population health intervention research using mixed methods. More broadly, successful implementation depends on the integration of the DMD into existing care pathways, clinical protocols and digital infrastructures (eg, electronic health records, hospital information systems).15–17 If the DMD is not embedded within routine workflows, even a technically well-performing system may fail to generate meaningful clinical benefit. As for drugs and other interventions, DMDs may face a lack of medical staff adherence and patient compliance (SOC). For example, a study evaluating the impact of an electronic sepsis surveillance system on mortality and length of stay in an emergency department reported no improvement in sepsis outcomes because of alert fatigue.18 Patients are also entitled to understand how the DMD works to make informed health decisions and know what happens if the DMD fails. Clinicians are expected to explain how drugs or interventions work. However, for AI-based DMDs that function like black boxes, it can be difficult to find measurable parameters of interpretability. A lack of adherence and compliance can also stem from fear of data security problems such as data leaks.14 A challenge of DMDs also concerns the validity of digital endpoints. For example, an application designed to assess mobility may not detect movement because it is switched off, malfunctioning or not being carried by the user. Patient-reported outcomes can also differ when data are gathered digitally versus in person.19 Finally, surrogate endpoints—defined as substitutes for the treatment effects of an intervention on the target outcome(s) of ultimate interest, measuring how patients feel, function or survive20—are widely used for DMDs to replace clinically relevant final endpoints, especially when the clinical effect requires long-term evaluation while manufacturers and users seek fast access to new technology. However, like drugs and other healthcare technologies, the relevance and validity of these surrogate endpoints are often questionable. Unlike drugs, but similar to non-pharmacological interventions (eg, medical devices, surgery, physiotherapy, psychotherapy), clinical outcomes for DMDs may depend on the training, competence, and the 3Us, that is, user, usage and intent to use. Indeed, once the analytical validation of the software embedded in the DMD was finalised on data, the translation to real-life setting is more complex. As observed in the early implementation of laparoscopic surgery, the efficacy of a procedure is also dependent on the context, setting and 3Us.21 Proposing clinical study designs under these specificities should be an important step for further usage and acceptance of these DMDs by patients and healthcare professionals. Hence, the operator is part of the intervention, which requires reporting operator characteristics (eg, experience), as explained in the Consolidated Standards of Reporting Trials (CONSORT) statement for randomised trials of non-pharmacological treatments.22 23 Indeed, taking medication is standardised. In contrast, the more experience the operator has with similar DMDs, the more effective the device will be. Consequently, two operators can have different performances, and one operator’s performance can improve over time. Examples of the challenges related to endpoints in the evaluation of AI-based DMDs are summarised in table 2. Table 2Examples of the challenges related to endpoints in the evaluation of AI-based DMDsKey challengesExamplesOrganisational impactProcess signals, sent by a user to a medical centre or doctor at any time, require fast human–machine interactions and connections.Lack of medical staff adherence and patient compliance Alert fatigue.19 Difficulty for clinicians to understand and explain to patient how AI-based DMDs that function like black boxes work.Fear of data security problems such as data leak.15 Validity of digital and surrogate endpoints21 An application designed to assess mobility may not detect movement because it is switched off, malfunctioning or not being carried by the user.Patient-reported outcomes can differ when data are gathered digitally versus in person.20 Relevance and validity of surrogate endpoints are often questionable while they are widely used for DMDs to replace clinically relevant final endpoints, especially when the clinical effect requires long-term evaluation while manufacturers and users seek fast access to new technology. Operator dependencyDepending on the training, competence and experience:Two operators can have different performances.One operator’s performance can improve over time. AI, artificial intelligence; DMD, digital medical device. Examples of the challenges related to endpoints in the evaluation of AI-based DMDs Alert fatigue.19 Alert fatigue.19 Difficulty for clinicians to understand and explain to patient how AI-based DMDs that function like black boxes work. Difficulty for clinicians to understand and explain to patient how AI-based DMDs that function like black boxes work. Fear of data security problems such as data leak.15 Fear of data security problems such as data leak.15 An application designed to assess mobility may not detect movement because it is switched off, malfunctioning or not being carried by the user. An application designed to assess mobility may not detect movement because it is switched off, malfunctioning or not being carried by the user. Patient-reported outcomes can differ when data are gathered digitally versus in person.20 Patient-reported outcomes can differ when data are gathered digitally versus in person.20 Relevance and validity of surrogate endpoints are often questionable while they are widely used for DMDs to replace clinically relevant final endpoints, especially when the clinical effect requires long-term evaluation while manufacturers and users seek fast access to new technology. Relevance and validity of surrogate endpoints are often questionable while they are widely used for DMDs to replace clinically relevant final endpoints, especially when the clinical effect requires long-term evaluation while manufacturers and users seek fast access to new technology. Two operators can have different performances. Two operators can have different performances. One operator’s performance can improve over time. One operator’s performance can improve over time. AI, artificial intelligence; DMD, digital medical device. Finally, endpoints for DMDs may differ across evaluation stages. During analytical validation, endpoints often correspond to technical performance metrics (eg, accuracy, discrimination), whereas clinical validation requires clinically meaningful outcomes. In addition, post-market evaluation may rely on real-world digital (RWD) indicators collected continuously. Ensuring coherence between these endpoints is also an important challenge. Study designs Whether self-modifying or not, a DMD (eg, software/algorithms) is a flexible technology, where modifications are easily shipped to the end user. To match this ease of modification, the clinical evaluation of DMDs calls for a change in the paradigm of clinical trials used to evaluate drugs. According to the French HTA body HAS (Haute Autorité de Santé) (2021),24 some specific characteristics of medical devices (compared with drugs) necessitate adapting traditional clinical trials. The European Union CORE-MD (Coordinating Research and Evidence for Medical Devices) project provided recommendations on the methodologies to be adopted for clinical investigations, but focusing on high-risk medical devices.5 The key points of this section for AI-based DMDs are presented in tables 3 and 4. Table 3Key characteristics, advantages and limitations of RCTs versus real-world studies for AI-based DMDsStudy designKey characteristicsAdvantagesLimitationsRCT Often parallel-group designs, sometimes crossover designsVarious control groups (often software without full functionality)Frequently not blindedSample sizes vary from a few to the thousands High internal validityGold standard for causal inference Time-consuming compared with DMD rapid evolutionDo not consider learning curveSmall samplesRelatively short-term follow-upLimited real-world applicabilityExpensiveMeasurement or surveillance bias, Hawthorne effect Issue of feasibility of a control/placebo group Real-world/observational studiesObservational studies, registries, routine data, etc. Monitor clinical evidence variations over time by considering the learning curve and across different usersTake into account the improvement of the device without repeating RCTsLarge and heterogeneous samplesLong-term follow-upCloser to real clinical practice (higher external validity)Better economic estimationLower costs34 + no Hawthorne effect Confounding bias Time-dependent confounding (individuals may use other interventions) More missing dataQuestionable validity of the digital evaluation criteria AI, artificial intelligence; DMD, digital medical device; RCT, randomised controlled trial. Key characteristics, advantages and limitations of RCTs versus real-world studies for AI-based DMDs Often parallel-group designs, sometimes crossover designs Often parallel-group designs, sometimes crossover designs Various control groups (often software without full functionality) Various control groups (often software without full functionality) Frequently not blinded Frequently not blinded Sample sizes vary from a few to the thousands Sample sizes vary from a few to the thousands High internal validity High internal validity Gold standard for causal inference Gold standard for causal inference Time-consuming compared with DMD rapid evolution Time-consuming compared with DMD rapid evolution Do not consider learning curve Do not consider learning curve Small samples Small samples Relatively short-term follow-up Relatively short-term follow-up Limited real-world applicability Limited real-world applicability Expensive Expensive Measurement or surveillance bias, Hawthorne effect Measurement or surveillance bias, Hawthorne effect Issue of feasibility of a control/placebo group Issue of feasibility of a control/placebo group Monitor clinical evidence variations over time by considering the learning curve and across different users Monitor clinical evidence variations over time by considering the learning curve and across different users Take into account the improvement of the device without repeating RCTs Take into account the improvement of the device without repeating RCTs Large and heterogeneous samples Large and heterogeneous samples Long-term follow-up Long-term follow-up Closer to real clinical practice (higher external validity) Closer to real clinical practice (higher external validity) Better economic estimation Better economic estimation Lower costs34 Lower costs34 Confounding bias Confounding bias Time-dependent confounding (individuals may use other interventions) Time-dependent confounding (individuals may use other interventions) More missing data More missing data Questionable validity of the digital evaluation criteria Questionable validity of the digital evaluation criteria AI, artificial intelligence; DMD, digital medical device; RCT, randomised controlled trial. Table 4Key characteristics, advantages and limitations of different types of comparators for AI-based DMDsComparator typeKey characteristicsAdvantagesLimitationsActive comparator Comparator with known clinical effectPreferred when available Ethically appropriate May require large sample sizeHarder to demonstrate superiority Sham Mimics treatment to maintain blindingCan replace the main treatment part with a fake therapy that looks effective or can remove contents assumed to be crucial for efficacy results. Controls for digital placebo effectEasier to present positive results Difficult to design a sham that looks like a plausible treatmentHighlights key intervention componentsUsing a sham means that the patient has all the ‘burden’ associated with using the device. Standard of care Usual carePreferred by industry Easier to show positive results compared with a competitorUnethical if an effective comparator existsControl that does not include technologyNo digital componentNo evolution over timeDigital placebo effectDMD as controlRequires the latest versionNo digital placebo effectMay evolve over timeAI, artificial intelligence; DMD, digital medical device; RCT, randomised clinical trial. Key characteristics, advantages and limitations of different types of comparators for AI-based DMDs Comparator with known clinical effect Comparator with known clinical effect Preferred when available Preferred when available May require large sample size May require large sample size Harder to demonstrate superiority Harder to demonstrate superiority Mimics treatment to maintain blinding Mimics treatment to maintain blinding Can replace the main treatment part with a fake therapy that looks effective or can remove contents assumed to be crucial for efficacy results. Can replace the main treatment part with a fake therapy that looks effective or can remove contents assumed to be crucial for efficacy results. Controls for digital placebo effect Controls for digital placebo effect Easier to present positive results Easier to present positive results Difficult to design a sham that looks like a plausible treatment Difficult to design a sham that looks like a plausible treatment Highlights key intervention components Highlights key intervention components Using a sham means that the patient has all the ‘burden’ associated with using the device. Using a sham means that the patient has all the ‘burden’ associated with using the device. Usual care Usual care Preferred by industry Preferred by industry AI, artificial intelligence; DMD, digital medical device; RCT, randomised clinical trial. Randomised clinical trials Randomised clinical trials (RCTs) are the gold standard for evaluating the causal relationship between an intervention and variation in the clinical conditions of patients. According to Huh et al,25 as for drugs, the clinical evaluation of DTx approved by the US Food and Drug Administration (FDA) is mainly realised using RCTs, often involving parallel-group designs, although crossover designs are also used. Nevertheless, unlike most drug trials, DTx trials can involve diverse control groups, often based on software without all its functions (see section ‘which comparator?’). Owing to DTx characteristics, trials are frequently unblinded, which creates bias (in measurement, classification or follow-up), a drawback also observed in the context of surgical innovation.26 Cluster randomisation can be an alternative to standard RCT, especially when organisational impact is evaluated. Nevertheless, the rapid development and evolution of DMDs compared with the usual duration of clinical trials makes their evaluation more complex than that of drugs. Indeed, the technology on which DMDs are based, especially machine learning or deep learning algorithms, has evolved rapidly. However, the design and execution of interventional trials are time intensive, often spanning from the initial planning phase to final patient inclusion and data analysis, which can take years. Furthermore, digital devices, particularly those based on AI, often collect large quantities of real-time data. These data may refine algorithms or adjust the device itself. Traditional evaluation methods do not consider this capacity for dynamic adaptation and continuous evolution of device performance, although some alternative designs, such as tracker trials, have been proposed in the context of changing technologies.27 Moreover, evaluation methods rarely account for the fact that the DMD must operate within complex digital ecosystems, interacting with other software systems, databases and clinical decision processes.15–17 Changes in interoperability, data standards or institutional protocols may indirectly modify the performance and impact of the DMD, even without explicit algorithmic updates. Which comparator? The choice of control or comparator group is crucial and depends on the study’s objective and the place of the DMD in the care strategy. An active comparator should be favoured if it exists.24 A DMD can also be assessed in addition to the reference treatment (‘on top’ or ‘add on’ trials). In all cases, equipoise must be observed. According to the FDA, a ‘sham’, which can be considered as a placebo control for a medical device, is ‘an ineffective device (or simulated procedure or possibly a drug or biological product) used under conditions designed to resemble the conditions of use under investigation as far as possible’.28 When a sham is designed, it must resemble a plausible treatment to keep patients blinded, which is often difficult for DMDs. Typically, sham controls are designed using different approaches. First, it is possible to replace the main treatment component with a fake therapy that looks effective. Second, contents assumed to drive efficacy can be removed. However, by observing which elements of digital interventions are removed, it becomes evident which characteristics are assumed to be most effective. Finally, using a sham still imposes the full ‘burden’ of device usage on the patient. In practice, Lutz et al 29 noted that, among DTx trials deploying psychosocial, cognitive or behavioural content, very different controls are used, from unblinded waitlists and treatment-as-usual designs to placebo-controlled trials. For example, industrial stakeholders prefer to compare their new DMDs (combined with the standard of care) to the standard of care alone rather than to a competitor. This approach often facilitates positive results and is like comparing a drug to a placebo rather than an active treatment. However, if the competitor has demonstrated efficacy, this becomes unethical, and the scientific question should shift from ‘should we use a DMD’ to ‘which DMDs should we use’? Doing so maintains equipoise. Other comparator-related issues are more specific to DMDs than to drugs. First, in an RCT comparing a new DMD to a widely used DMD, the learning curve for the new DMD may highlight differences in user experience more than differences in clinical outcomes. This problem was illustrated by the conventional versus laparoscopic-assisted surgery in patients with colorectal cancer trial for laparoscopic colorectal surgery.30 Thus, the operator’s learning curve must be considered by, for example, using a training time. Expertise-based randomised controlled trials can also be envisaged, or the analysis can be adjusted to account for the learning effect, using, for example, hierarchical Bayesian models.31 Another issue is that studies using a control without technology often yield larger effect sizes, which is called the digital placebo effect.32 In such cases, DMD benefits may be inflated by patient expectations, novelty effects or increased engagement with the technology. Many DMDs are complex multimodal interventions, making it difficult to isolate the active component and determine an appropriate comparator. Finally, since DMDs are frequently updated, comparisons must involve recent versions, not obsolete versions (see figure 1). An interesting question concerns the comparator’s own evolution over time, especially if it is an AI-based DMD, as such systems can continuously improve through updates, retraining or new data integration. Unlike traditional medical interventions, where the comparator remains stable throughout a trial, AI-based comparators may evolve during the study period, altering performance and clinical relevance. This dynamic nature raises significant methodological trial design challenges, as well as regulatory considerations regarding how to ensure fair and meaningful comparisons over time. Real-world studies Observational studies using RWD can provide additional information compared with RCTs. The US business and regulatory environments are highly conducive to the use of RWD and have a longer history of leveraging RWD compared with Europe. These advantages stem from two key factors: a private sector, where numerous large, data-driven health technology companies have access to extensive patient-level data, and a progressive regulatory framework, with initiatives supporting real-world evidence dating back to 2018. Tarricone et al 33 emphasised the value of real-world studies in generating clinical evidence to evaluate medical devices, noting advantages, such as (1) monitoring clinical evidence variations over time, capturing learning curves and user variability; (2) accounting for device improvement without repeating RCTs; (3) including large, heterogeneous samples; (4) enabling long-term follow-up; (5) providing results closer to real-world practice, thus allowing higher external validity; (6) providing better economic estimation and (7) being less expensive. Compared with drugs, the first two advantages are specific to medical devices. Another advantage is the absence of the Hawthorne effect, which in psychology describes the situation in which the results of an experiment are not due to experimental factors but rather to subjects who are aware of participating in an experiment in which they are being tested, which translates into greater motivation. Moreover, for example, DTx could be readily evaluated in home-based settings and delivered through the Internet.25 Other evaluation approaches include retrospective analyses using insurance claims or usage data, which enable long-term evaluation of effectiveness. Digital biomarkers obtained from real-time and continuous log data are also used to improve the objectivity of endpoints. The growing reliance on registries and post-market surveillance infrastructures, as emphasised in the CORE–MD project, illustrates the increasing importance of structured RWD systems for the evaluation of high-risk medical devices.5 34–36 Registries may provide continuous performance monitoring, support signal detection and complement premarket trials, particularly in the context of iterative device modifications. The main drawbacks of real-world studies are biases, mainly indication bias (related to confounding), where treatment allocation is influenced by patient characteristics (such as better health, attraction to new technologies) and clinician experience. Therefore, observational studies can be limited in terms of the imputability of effects. Nonetheless, methods exist to correct for this bias and time-dependent confounding bias, such as propensity scores37 or instrumental variables, G-computation, marginal structural models38 and the target trial framework.39 Challenges in using RWD for DMDs also include missing data and the questionable validity of digital evaluation criteria. These issues may also arise in RCTs. However, small DMD adaptations can be performed continuously, making it unrealistic to conduct RCTs for every single change. On this subject, the HAS mentions the possible acceptance of the evolutivity of a DMD in a study if the main functionalities are not transformed.6 The RWD collected throughout the DMD’s life cycle can assist in adjusting conclusions. When to evaluate? The drug development process is long and structured, including well-defined clinical phases. Once approved, the drug rarely changes. In contrast, the development of a DMD is often fast, and a DMD is often updated with new versions, even after market release. Indeed, a DMD requires continuous updates to ensure compatibility with operating systems and browser software, enhance performance, address bugs and introduce new modules that provide additional benefits to patients. However, defining what constitutes a ‘new version’ of a DMD is itself a challenge since updates may range from minor technical adjustments (eg, bug fixes, interface changes) to substantial modifications affecting algorithms, data inputs or intended use. The definition of a new version may only be assessable in retrospect: it could be any change leading to a change in the output that impacts patient care. Furthermore, for AI-based DMDs developed using continuous learning systems, version boundaries may become blurred, as performance may evolve without a clearly identifiable software release. Therefore, there is no fixed version of the algorithm to evaluate. Consequently, each new update and evolution of the DMD (based on continuous data learning or algorithm evolution) cannot be systematically re-evaluated and validated in a clinical study. Selecting when to perform the clinical evaluation is a first key point. It is advisable to carry out this evaluation as early as possible, without forgetting to mention, in the protocol, the possible changes to the medical device over time that cannot alter its main function. This requires explicit version control procedures, including documentation of software iterations, algorithm modifications and their expected clinical impact. The next step is to define what, in the evolution of the medical device, should lead to a new evaluation and what can be considered an adjustment of the medical device ‘at the margin’. Distinguishing between minor updates and meaningful modifications is particularly complex for AI-based DMDs, where small algorithmic changes may have unpredictable effects on performance. This question is certainly DM-dependent. According to the HAS recommendations on the evolution of connected devices,40 the need for a new study depends on the fact that extrapolation from one version to another is no longer possible. However, in the case of DMDs, the evaluation of extrapolation from a learning algorithm is much more complex. Thus, it is necessary to develop extrapolation and prediction approaches indicating if and when a new clinical study will be needed to optimise and plan without pausing updates for too long. When a new clinical evaluation is necessary, we must decide how to carry it out and with what data (see illustration in figure 1). Finally, like pharmacovigilance for drugs, DMDs need to be reassessed constantly, which calls for a lifecycle-based evaluation framework integrating version tracking, post-market monitoring and predefined criteria triggering re-evaluation. Unlike drugs, updates to a DMD may not only change its intrinsic performance but also its integration within care processes and digital ecosystems.15–17 A modification in the algorithm, user interface or alert system may have ripple effects on clinician workload, decision-making pathways, patient behaviour and data flows across interconnected systems. Therefore, evaluation frameworks should anticipate and monitor these cascade effects, ensuring that updates do not unintentionally disrupt care protocols or generate new safety risks. In the European regulatory framework, post-market clinical follow-up (PMCF) is mandatory under the Medical Device Regulation and forms part of the continuous clinical evaluation process. PMCF aims to proactively collect and evaluate clinical data on a device once it is CE-marked, in order to confirm its safety and performance, identify emerging risks and ensure that the benefit–risk balance remains favourable throughout its lifecycle. For DMDs, PMCF may rely on RWD, registries, usage logs or specifically designed post-market studies. Given the iterative and software-based nature of DMDs, PMCF is not only a regulatory obligation but also a methodological necessity to monitor performance drift, unintended consequences of updates and changes in user behaviour over time. In this context, PMCF can serve as a structured feedback loop between real-world deployment and ongoing clinical evaluation. Discussion Compared with interventions based on drugs and even other medical devices or surgical procedures, DMDs present specific characteristics. First, the development of the device, which could be standalone software, can be very quick, performed by startup companies requiring a rapid return on investment. This rapid development has led health authorities to allow fast-track processes as proposed by the FDA (https://www.fda.gov/patients/fast-track-breakthrough-therapy-accelerated-approval-priority-review/fast-track). Furthermore, the algorithms developed are prone to change quickly, with improvements in terms of accuracy or broadness of applicability. Therefore, rather than being fixed, algorithms can change over time in a so-called continual learning framework.41 However, the improvement is not obvious, and in contrast, the performance of the algorithm could be deteriorated by the addition of new biased and noisy data. The potential change of the intervention during an evaluation process is unique for this type of setting and raises new challenges for their evaluation. Chatbots employing large language models with reinforcement learning are a caricatured example.42 However, COVID-19 vaccines offer a counterexample with some similarities to DMDs with respect to the need for rapid development and adaptation due to virus variants.43 Beyond algorithmic evolution, the integration of DMDs into care pathways and digital ecosystems introduces system-level dependencies, where updates may have indirect effects on clinical protocols, interoperability and healthcare organisation.15–17 Addressing these cascade effects requires moving from a device-centred evaluation paradigm to a system-level perspective. Although some of the evaluation methods used for drugs are applied to DMDs, specific issues make these methods inappropriate. These issues are consistent with broader reflections on the evaluation of high-risk medical devices, notably those highlighted by the European CORE–MD project, which focuses on reviewing methodologies of clinical investigations, advising on study designs and developing recommendations for aggregating clinical data from registries and other real-world sources.34 Our analysis extends these reflections to the specific case of DMDs, where algorithmic evolution and data dependency further increase methodological complexity. To assess DMDs appropriately, more flexible approaches must be developed, such as adaptive trial designs or assessment methods based on real-time data. Currently, no formal or specific recommendations exist for clinical trial designs. However, continuous learning implies continuous clinical evaluation and therefore continuous corrections and adaptations. Thus, innovative clinical trial designs must be proposed. With respect to the challenge of evaluating a new DMD in the various dimensions needed (technical, clinical, organisational impact), simulations could be interesting to maximise the success of prospective studies. Simulations, also called in silico trials44 or digital twins, constitute a framework for modelling the effect of the intervention using the DMD. This type of approach has been used for the evaluation of complex interventions, such as population cancer screening.45 Owing to their continuous evolution, AI-based DMDs could more closely approach human-based interventions, such as those studied in the field of population health intervention research. The Medical Research Council (MRC) guidance on complex interventions provides a useful framework,46 as it acknowledges that interventions often adapt to their context and evolve throughout the evaluation process, much like AI-based DMDs, which evolve with new data and software updates. In particular, the MRC emphasises the importance of iterative and context-specific evaluation and the use of complementary methodologies and outcomes to evaluate the effectiveness of complex interventions. It defines the core elements to consider at each step of intervention development: consider context; develop, refine and retest programme theory; engage stakeholders; identify key uncertainties; and refine the intervention and economic considerations. Adopting certain principles could improve the clinical validation framework for DMDs.