Hospital-based real-world evidence in health technology assessment: insights from a scoping review of European, Australian and North American guidance and expert interviews Authors: Zainab Al-khayat, Nora Franzen, Valesca P Retèl, Wim H Van Harten Published: 2025 ABSTRACT Abstract AbstractObjectivesHospital data can inform decision-makers with real-time evidence, yet it remains underutilised. This study aims to compare international health technology assessment (HTA) and regulatory real-world evidence (RWE) guidance, focusing on their applicability to hospital data.Study design, setting and participantsWe used a two-step sequential qualitative design: a scoping review and semi-structured interviews with HTA experts. We searched for RWE guidance for HTA in 12 countries: the UK, Germany, Italy, Spain, France, Finland, the Netherlands, Portugal, Denmark, Canada, the US and Australia, along with the European Medicines Agency and Food and Drug Administration. The expert interviews aimed to validate document selection and assess their applicability to hospital data. We analysed the interviews thematically.ResultsWe identified 19 guidance documents providing recommendations for RWE. Of these, four documents explicitly provided recommendations tailored to hospital data, while two others did so implicitly. The scope, definition and applications of RWE vary among guidance. Recommendations across all agencies mainly address the clinical-effectiveness domain, with limited guidance on quality-of-life and patient-reported outcomes, and none on real-world cost. The interviews identified seven themes playing a role in using hospital data: data-related, generalisability, ethical/legal, organisational, communication, governance and technology-related. Barriers included data availability, access, timeliness, quality, validation and heterogeneity. HTA experts emphasised the need for standardised policies.ConclusionsThere is a lack of harmonisation in assessing RWE among HTA and regulatory agencies. The available RWE guidance documents provide limited guidance on real-world hospital data. Considering their unique nature and to unlock their potential for HTA, we emphasise the need for more in-depth guidance tailored to the hospital context.Keywords: HEALTH ECONOMICS, Hospitals, Electronic Health Records STRENGTHS AND LIMITATIONS OF THIS STUDY The study followed a rigorous methodology, applying Arksey and O’Malley’s framework and reporting in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for Scoping Reviews Checklist. The study followed a rigorous methodology, applying Arksey and O’Malley’s framework and reporting in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for Scoping Reviews Checklist. The focus on health technology assessment (HTA) and regulatory body websites allowed for a targeted assessment of guidance documents directly relevant to decision-making. The focus on health technology assessment (HTA) and regulatory body websites allowed for a targeted assessment of guidance documents directly relevant to decision-making. Semi-structured expert interviews were conducted to enrich the findings and ensure practical relevance. Semi-structured expert interviews were conducted to enrich the findings and ensure practical relevance. Guidance documents published after July 2024 are not included and may reflect more recent developments. Guidance documents published after July 2024 are not included and may reflect more recent developments. While the focus on European HTA and regulatory sources ensured policy relevance, it may have excluded insights from broader international, academic or grey literature. While the focus on European HTA and regulatory sources ensured policy relevance, it may have excluded insights from broader international, academic or grey literature. Introduction Real-world data (RWD) refer to health outcomes, including benefits, risks and resource use, collected during routine clinical practice rather than through traditional randomised controlled trials (RCTs).1 RWD includes clinical and economic outcomes, patient-reported outcomes (PROs) and health-related quality of life (HRQoL), obtained from sources such as patient registries, observational studies and electronic health records (EHRs).1 Real-world evidence (RWE) is the evidence derived from analysing RWD.2 The healthcare sector generates 30% of global data, with an expected annual growth of 36% by 2025.3 The growth is driven by the increased use of EHRs, the expanding scope of laboratory and imaging data, and a broad range of medical technology advancements.4 These factors contribute to the rich data systems and sources related to healthcare provision, creating growing and innovative opportunities for using RWD.5 Proper use of RWD can enhance treatment personalisation and show the efficacy of health technologies in real-world contexts.5 RWE application has evolved from focusing on safety and post-market analysis to a more involved role across the product lifecycle.5 Health technology assessments (HTAs) have seen a rise in using RWE from 6% in 2011 to 39% in 2021, primarily supporting epidemiology (39%), safety (39%), effectiveness (33%), utilities (14%) and treatment costs (10%).6 This growth reflects growing acceptance and use of RWE in HTA.6 Despite their potential, RWD sources are not uniformly used in HTA.7 8 While hospital data, including EHRs, imaging, financial data, pathology and laboratory reports, can provide real-time insights, as seen during the COVID-19 pandemic, they are not fully leveraged in HTA submissions compared with registry data (6% vs 49%, respectively).6 9 Registry data are preferred for their structured nature; however, their strict objectives often limit their comprehensiveness.10 Whereas EHRs, despite their greater comprehensiveness, are underutilised in HTA submissions due to their less structured data and potential bias concerns.6 10 11 Nevertheless, the evolution of methodological approaches to reduce bias and the technological advancements, such as natural language processing techniques, have improved the overall credibility and reliability of hospital data.12,15 HTA agencies differ in their use, assessment and acceptance of RWE, influenced by their defined evidence requirements and stakeholder views.7 16 Although HTA agencies show consistency in certain assessment criteria, such as evaluating the clinical effectiveness and safety, there are considerable variations in economic assessment criteria across countries. The UK, France, Germany, Australia and Canada favour a payer perspective, whereas Norway, Portugal, Denmark, Spain, Finland, Sweden and the Netherlands adopt a broader societal viewpoint.17 18 Belgium and Italy recommend considering both perspectives.17 18 All these heterogeneities hinder comparability; for instance, submissions using the same RWE from the same sources received varied responses across different agencies.19 20 Consequently, the opportunities for RWE in technology assessments remain unclear and inconsistent across HTA bodies.21 Recognising the growing importance of RWE in technology assessment and rapid technological advancements, many HTA and regulatory agencies have recently issued guidance documents on using RWE. Due to variations in scope and evidence criteria, different agencies may have differing perspectives on RWE. The objective of this paper is to comprehensively compare international HTA and regulatory agencies’ RWE recommendations, with a special focus on how these recommendations apply to real-world hospital data. Methods The study employed a two-step qualitative research design: a scoping review and semi-structured interviews with country representatives. Starting with a European focus, twelve countries were selected to offer a broad international perspective. This includes EU countries involved in the ONCOVALUE consortium (Finland, the Netherlands, Portugal, Italy and Denmark), as well as the UK, Spain, Germany and France, countries whose HTA bodies are influential in the European landscape. The United States (US), Canada and Australia were also included due to the global influence of their HTA. We further included the European Medicines Agency (EMA) and the Food and Drug Administration (FDA), both influential in RWE development from a regulatory perspective. Scoping review The review was conducted using the framework of Arksey and O’Malley for scoping studies.22 The preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (PRISMA-ScR) was followed and used to report the study.23 For scoping the relevant documents, we followed the research question: ‘How do international HTA and regulatory agencies differ in their guidance on the use of RWE, and to what extent do they provide specific recommendations for incorporating real-world hospital data into health technology assessment?’ In this context, ‘guidance’ refers to official documents issued by these agencies that outline overarching principles or methodological approaches. Within these documents, recommendations denote specific, formal suggestions advising a particular course of action or decision. Identifying relevant studies To determine the most appropriate approach for addressing our research questions, we conducted a preliminary search of electronic literature databases to assess the feasibility of using published studies. While some publications discuss RWE guidance documents, they did not provide sufficient detail on the content of these documents to address our research question. Therefore, we focused on the websites of HTA and regulatory agencies, where such guidance documents are typically published. We conducted a scoping review searching these websites to identify guidance documents on RWE use in technology assessments (online supplemental table 1). We applied a set of RWE-related keywords to locate potentially relevant documents. The full search strategy is provided in online supplemental table 2. Study selection We followed the predefined inclusion and exclusion criteria to guide the selection process. Included guidance documents explicitly provided recommendations on the use of RWE in HTA or regulatory contexts and were published between January 2015 and July 2024. We excluded commentaries, news articles, documents older than 2015, previous versions that were updated, guidance documents targeting only registry data, and those already covered within a more comprehensive guidance document. All results were gathered in an Excel document, and duplicates were removed. One author (ZA) screened all titles and content using predefined inclusion criteria, while a second author (NF) independently screened 10% of the records to ensure reliability. Discrepancies were resolved through discussion with the last author (WvH). Charting the data A structured data extraction form was created to extract relevant information from the selected guidance documents, which included the categories: publishing entity, publication year, country of origin, binding status (eg, binding or non-binding), scope of the document, definitions of RWD, specific recommendations for using RWE, RWE applications, and the use of RWE in the initial and re-assessment submissions, which we refer to as RWE acceptance in this paper. For RWE applications, we considered all potential uses of RWE as stated in the documents. Collating, summarising and reporting the results We adapted the approach used by the Canadian Agency for Drugs and Technologies in Health (CADTH) to categorise all the identified recommendations into three domains: planning, conducting and reporting RWE.24 For each, we developed subcategories, including data source and data specification. These subcategories represented the topics to which each recommendation was mapped (see online supplemental table 3). It is important to note that we only considered explicitly stated recommendations. We assessed how these guidance documents address HTA and regulatory key domains, including clinical effectiveness, safety, quality of life (QoL) and cost. We compared HTA and regulatory guidance documents across the clinical-effectiveness and safety domains due to their shared relevance, and included the QoL and cost domains for HTA guidance documents only. Additionally, we examined the guidance documents for source-specific recommendations, focusing on hospital data. This includes both recommendations that explicitly reference hospital-based data sources and those addressing the specifications of certain data types, such as unstructured data, free-text mining and longitudinal data. HTA expert interviews We interviewed HTA experts from all included countries to validate the guidance selection and to obtain feedback on their feasibility. By feasibility, we mean the practicality and likelihood of successfully implementing the recommendations provided in the guidance documents. A predefined, semi-structured interview guide was developed and used for all the interviews (online supplemental file 1). The interviewees were asked whether the available guidance documents apply to hospital data, whether a source-specific guidance is needed, and about the barriers and facilitators specifically related to using hospital-based RWD for HTA. Interviewees were selected using purposive sampling, considering experts related to or knowledgeable on the topic, with at least 5 years of experience in assessing, performing or conducting HTA. The interviews, conducted between October 2023 and June 2024, were held in a hybrid format (online or in person) and subsequently transcribed and thematically analysed using the Braun and Clarke method.25 The transcripts were de-identified, with sensitive content replaced by neutral insertions. The participants were informed about the purpose, content and procedures of the study, and both written and verbal informed consent were obtained prior to the interviews. The Institutional Review Board of the Netherlands Cancer Institute granted ethical approval to this study (IRBd23-246). Patient and public involvement statement Patients or the public were not involved in the design, conduct, reporting or dissemination plans of our research. Results In the following, we present the results of the scoping review and the interviews. Scoping review The search identified a total of 3985 records. After removing duplicates, 1731 records remained. Title screening led to the exclusion of 1655 records, leaving 76 for full-text screening. Following full-text screening, 59 records were excluded lacking RWE recommendations, leaving 17 guidance documents for analysis (figure 1). Among these, seven documents are issued by HTA agencies, while ten are issued by the regulatory agencies, with the EMA contributing one and the FDA contributing nine. Additionally, two guidance documents were included based on the interview results: one from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and another from the European Network for Health Technology Assessment (EUnetHTA).9 26 In total, we included 19 guidance documents: seven from HTA agencies, ten from regulatory agencies and two from ISPOR and EUnetHTA, all explicitly providing recommendations on RWE use in health technology assessments. Of these, four guidance documents (issued by three agencies) explicitly provide recommendations tailored to hospital data,927,29 while two others do so implicitly.24 30 All included documents are in English, as in table 1. PRISMA flow diagram showing the process of identification, screening and inclusion of documents. AGENAS, Agenzia Nazionale per i Servizi Sanitari Regionali; AQuAS, Agència de Qualitat i Avaluació Sanitàries de Catalunya; CADTH, Canadian Agency for Drugs and Technologies in Health; FDA, Food and Drug Administration; FinCCHTA, Finnish Coordinating Center for Health Technology Assessment; HAS, Haute Autorité de Santé; ICER, Institute for Clinical and Economic Review; INFARMED, Autoridade Nacional do Medicamento e Produtos de Saúde; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; Medicinrådet, Danish Medicines Council; NICE, National Institute for Health and Care Excellence; PBAC, Pharmaceutical Benefits Advisory Committee; RWE, real-world evidence; ZiN, Zorginstituut Nederland Table 1An overview of RWE guidance documents identifiedBodyCountryEntityRWE Guidance*Guidance on RWE hospital dataPublish dateHTA bodyCACADTH24YesYes (Implicitly)May-23DEIQWiG27YesYes (Explicitly)Jan-20DKMedicinrådet31YesNoJun-23ESAQuAS36YesNoJan-22FRHAS32YesNoJun-21UKNICE30YesYes (Implicitly)‡Jun-22USICER33YesNoMar-18AUPBACNo†––FIFinCCHTANo––ITAGENASNo––NLZiNNo–PTINFARMEDNo––EUnetHTAEUEUnetHTA26YesNoMar-22ISPORINTISPOR9YesYes (Explicitly)Apr-24RegulatoryEUEMA34YesNoApr-24USFDA2829 35 37,42Yes (nine documents)Yes (Explicitly)(Aug-2017- Jul-2024)* The identified guidance documents are all in English†According to our interviews, Australia is developing RWE guidance.‡updated June-23AGENAS, Agenzia Nazionale per i Servizi Sanitari Regionali; AQuAS, Agència de Qualitat i Avaluació Sanitàries de Catalunya; AU, Australia; CA, Canada; CADTH, Canadian Agency for Drugs and Technologies in Health; DE, Germany; DK, Denmark; EMA, European Medicines Agency; ES, Spain; EU, European Union; EU, European Union; EUnetHTA, European Network for Health Technology Assessment; FDA, Food and Drug Administration; FI, Finland; FinCCHTA, Finnish Coordinating Center for Health Technology Assessment; FR, France; HAS, Haute Autorité de Santé; ICER, Institute for Clinical and Economic Review; INFARMED, Autoridade Nacional do Medicamento e Produtos de Saúde; INT, international; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; IT, Italy; Medicinrådet, Danish Medicines Council; NICE, National Institute for Health and Care Excellence; NL, The Netherlands; NR, Not reported; PBAC, Pharmaceutical Benefits Advisory Committee; PT, Portugal; UK, United Kingdom; US, United States; ZiN, Zorginstituut Nederland. An overview of RWE guidance documents identified The identified guidance documents are all in English According to our interviews, Australia is developing RWE guidance. updated June-23 AGENAS, Agenzia Nazionale per i Servizi Sanitari Regionali; AQuAS, Agència de Qualitat i Avaluació Sanitàries de Catalunya; AU, Australia; CA, Canada; CADTH, Canadian Agency for Drugs and Technologies in Health; DE, Germany; DK, Denmark; EMA, European Medicines Agency; ES, Spain; EU, European Union; EU, European Union; EUnetHTA, European Network for Health Technology Assessment; FDA, Food and Drug Administration; FI, Finland; FinCCHTA, Finnish Coordinating Center for Health Technology Assessment; FR, France; HAS, Haute Autorité de Santé; ICER, Institute for Clinical and Economic Review; INFARMED, Autoridade Nacional do Medicamento e Produtos de Saúde; INT, international; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; IT, Italy; Medicinrådet, Danish Medicines Council; NICE, National Institute for Health and Care Excellence; NL, The Netherlands; NR, Not reported; PBAC, Pharmaceutical Benefits Advisory Committee; PT, Portugal; UK, United Kingdom; US, United States; ZiN, Zorginstituut Nederland. Scope, definitions and the role of RWE in technology assessments The reviewed guidance documents showed considerable heterogeneity in scope, terminology and the stated role of RWE in HTA. All 19 documents define their in-scope area, while only 11 specify what falls outside their scope. Most frequently, RWE studies are defined as in-scope (reported by six agencies),2426 30,33 while traditional RCTs are defined as out-of-scope (reported by five agencies).27 30 32 34 35 Among the remaining agencies, several also consider RWE studies within their scope but describe them using alternative terminology or study types, such as observational or descriptive studies (see online supplemental table 4). The Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (IQWiG) applies the narrowest scope, focusing on comparative studies such as target trial emulation, pragmatic trials and adjusted indirect treatment comparisons with a common comparator, excluding single-arm trials.27 By contrast, the Catalan Agència de Qualitat i Avaluació Sanitàries de Catalunya (AQuAS) and EMA adopt a broader scope that includes observational or non-interventional studies.30 36 The FDA’s nine guidance documents span study designs (eg, non-interventional), data sources (EHRs, claims and medical devices), and study types (eg, safety, post-marketing).2829 35 37,42 Only three guidance documents, two issued by the FDA and one by ISPOR, explicitly specified hospital data, particularly EHRs, as within the scope for health technology assessments,9 28 29 whereas IQWiG makes only brief reference to EHRs.27 We found inconsistent terminology related to RWE studies across the agencies. The guidance documents use overlapping but not always synonymous terms, such as prospective, observational/non-interventional, predictive, descriptive and comparative studies. All but one document defines RWD; however, the definitions vary, with most emphasising the importance of reflecting real-world clinical practice, patient perspectives and healthcare delivery (see online supplemental table 5). A similar variation is observed in the definitions of pragmatic trials, which differ in their emphasis on trial flexibility and links to real-world conditions, as seen across IQWiG, the Haute Autorité de Santé (HAS) and the National Institute for Health and Care Excellence (NICE); however, all consistently highlight randomisation as an essential feature.27 30 32 When looking at the role of RWE in technology assessments, all guidance documents recognise that RWE can help address evidence gaps, but its stated applications and acceptance vary. The most frequently mentioned application of RWE is assessing treatment effectiveness (reported by six agencies),2428,32 34 35 37 followed by its use in epidemiology, describing patient demographics and resource use, providing external control arms, or supporting safety studies (see online supplemental table 6). Acceptance also differs between pre-authorisation (initial assessment) and post-authorisation (re-assessment) phases: seven agencies accept RWE in re-assessment (HAS, the Institute for Clinical and Economic Review (ICER), IQWiG, NICE, EUnetHTA, EMA and FDA),26,3032 while only five do so in initial assessments (HAS, ICER, NICE, EUnetHTA and FDA).26 30 32 33 37 40 FDA and EUnetHTA accept RWE for new indications of already approved medicines.26 29 IQWiG does not report on using RWE in the initial phase, and EMA’s position remains unclear, stating that “Their use for assessing medicines’ efficacy is hindered by methodological limitations.”34 Guidance documents issued by AQuAS, CADTH and ISPOR do not discuss the use of RWE in either the initial or re-assessment phase, and Medicinrådet’s position remains unclear (see online supplemental table 7). RWE guidance: recommendations specific to hospital data Hospital data are limitedly addressed in current RWE guidance documents. Of the eleven agencies, six make no reference to hospital data. The remaining five provide some recommendations, though with substantial differences in scope and depth: four guidance documents (two from the FDA, IQWiG and ISPOR) explicitly address hospital data,927,29 whereas NICE and CADTH refer to it only implicitly.24 30 Where hospital data are explicitly discussed, the focus is largely on the quality and reliability of data from EHRs. The FDA provides the most detailed documents, requiring not only complete, accurate and relevant data but also explicit assessment of missing data, including their impact on study outcomes, the reasons for missingness (for example, not collected vs not recorded), and appropriate handling through methods such as imputation or sensitivity analyses. ISPOR similarly emphasises completeness and reliability, covering structured, unstructured and derived data, and provides a SUITABILITY checklist to assess the fit-for-purpose of EHRs for HTA. The checklist places greater emphasis on documentation, traceability, interoperability and fitness for purpose rather than on statistical handling of missing data. IQWiG shares the focus on representativeness, inclusion and exclusion criteria, and confounders or effect modifiers, providing only limited recommendations on EHR use and none on handling missing data derived from EHRs. In contrast, NICE and CADTH mainly address the transformation of unstructured hospital data into structured formats through human abstraction, algorithms or artificial intelligence (AI), but offer limited methodological detail. Both NICE and the FDA highlight the need for clear conceptual and operational definitions of key variables, although only the FDA provides instructions on how to develop them. Across all these guidance documents, transparency, traceability and reliability of data are consistently emphasised, yet the level of detail and the aspects covered differ widely between agencies. Most guidance documents focus narrowly on EHRs, leaving other types of hospital-based data, such as cost data linked to treatments and care pathways, unaddressed. RWE for HTA: what guidance is provided along the clinical-effectiveness, QoL and costs domains While all documents aim to guide the use of RWE in HTA, the level of detail and comprehensiveness varies both across guidance documents and across the HTA domains of clinical-effectiveness, QoL and costs. Table 2 summarises the number of recommendations by theme across all guidance documents. Our analysis shows that guidance documents span the entire RWE study cycle: planning and design (89 recommendations), study conduct with attention to data sources (115), data specification (337), population, intervention, comparator, outcomes, time (PICOT) formulation (157), patient-related outcomes (21), use of advanced technologies (11), ensuring robustness (75), and the reporting phase, covering analysis and interpretation (92), dissemination (12) and study limitations (13). Table 2Comparison of recommendations by topic across RWE guidance documentsThemeSubtheme (N)*TopicsNICEAquasHASMedici-nrådetIQWiGCadthIsporEUnet-HTAIsporEMAFDATotal n=154Total n=98Total n=81Total n=48Total n=109Total n=213Total n=23Total n=10Total n=44Total n=87Total n=503RWE planningResearch Questions and Study Design (89)Research question 5 5 344431113Study design79349611–10 14 Feasibility assessment11––––––– 12 11Protocol4352631––1 20 RWE conductingData sources (115)Data source – General20206510142–125 40 Multiple sources5–––51––15 10 Across-border sources––––1 6 ––––1Data Specifications (337)Data dictionary – General21–11 5 1–11 5 Conceptual/operational definitions1––––––––– 35 Data access111––3––22 5 Data collection422–54–––2 20 Data quality86232161156 51 Data curation31–1–41–32 28 Data relevance – General 4 –211–––111Data comparability for external control arm–––––––––– 11 Data validity241136111– 24 Data linkage121–25–––1 7 Data management311–1–––31 38 Data protection412––1––1– 14 PICOT (157)Selection Criteria22214 12 11–55Study participants45–1– 9 –––13Exposures242215–––1 21 Comparators–11–2 6 –––23Exposure/comparator11–113––1– 9 Outcomes104211 27 121126Time zero2 3 –1–1––––1Follow-up 4 111–2––2– 4 RWE conductingPatient-related outcomes (21)PROs and QoL–– 18 ––1––––2Adverse event studies1–1–1 3 ––––1Pregnancy studies–––––––––– 7 Advanced technology (11)General–––––––––– 3 Machine learning1– 2 –––––––1Data federation 1 ––––––––––Common data models 3 ––––1––2–2Robustness (75)Bias825416–212 13 Missing data324112–––1 9 Confounding76228 10 –––56Effect modifier21––1 4 –––22Sensitivity analyses11––24–––3 6 RWE reportingAnalysis and interpretation (92)Analyses and statistical reporting1213311 16 4–16 16 Statistical significance/ estimates precision 5 31––4––31 5 Reporting results71375 15 31–22Generalisability and representativeness111112––11 4 Dissemination (12)Dissemination plan–21–––3––– 9 Study limitations (13)Data related––––– 3 ––––2Bias related––––– 3 ––– 3 1Others112–– 3 –––11Only explicit information is considered. Bold highlighted numbers indicate the highest number of recommendations provided within a single guidance document for that topic.*N in the Subtheme column represents the total number of distinct recommendations across all guidance documents. This number may not equal the sum of the rows, as some recommendations appear in multiple documents. The total N represents the total number of recommendations included in each guidance document.AQuAS, Agència de Qualitat i Avaluació Sanitàries de Catalunya; CADTH, Canadian Agency for Drugs and Technologies in Health; EMA, European Medicines Agency; EUnetHTA, European Network for Health Technology Assessment; FDA, Food and Drug Administration; HAS, Haute Autorité de Santé; ICER, Institute for Clinical and Economic Review; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; Medicinrådet, Danish Medicines Council; NICE, National Institute for Health and Care Excellence; PICOT, Population, Intervention, Comparator, Outcomes, Time; PROs, Patient-Reported Outcomes; QoL, Quality of Life. Comparison of recommendations by topic across RWE guidance documents Only explicit information is considered. Bold highlighted numbers indicate the highest number of recommendations provided within a single guidance document for that topic. N in the Subtheme column represents the total number of distinct recommendations across all guidance documents. This number may not equal the sum of the rows, as some recommendations appear in multiple documents. The total N represents the total number of recommendations included in each guidance document. AQuAS, Agència de Qualitat i Avaluació Sanitàries de Catalunya; CADTH, Canadian Agency for Drugs and Technologies in Health; EMA, European Medicines Agency; EUnetHTA, European Network for Health Technology Assessment; FDA, Food and Drug Administration; HAS, Haute Autorité de Santé; ICER, Institute for Clinical and Economic Review; IQWiG, Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; ISPOR, International Society for Pharmacoeconomics and Outcomes Research; Medicinrådet, Danish Medicines Council; NICE, National Institute for Health and Care Excellence; PICOT, Population, Intervention, Comparator, Outcomes, Time; PROs, Patient-Reported Outcomes; QoL, Quality of Life. The analysis demonstrated differences in the amount of detail and comprehensiveness of recommendations, with the FDA providing the highest number of specific recommendations and EUnetHTA the fewest. Most recommendations relate to data specifications, such as data quality, although the extent of coverage varies by agency, with the FDA and IQWiG providing most recommendations. Similar patterns are seen across other topics, with some addressed by only a few guidance documents. For example, only the EMA, FDA, NICE and HAS discuss conducting a feasibility assessment prior to the study protocol to evaluate the suitability of data sources for answering the research question(s); HAS refers to this step as a ‘protocol synopsis.’29 30 32 34 35 39 42 In addition, only the FDA considers specific populations often excluded from clinical trials, such as pregnant and lactating individuals, infants, children, adolescents, the elderly, immunocompromised patients and those with disabilities or rare disorders.35 It provides recommendations on the use of secondary data in pregnancy studies for post-marketing safety assessments, specifying time variables such as conception and pregnancy dates, trimester and gestational age. The analysis across all three HTA domains (clinical effectiveness, cost and QoL) showed that, regarding the QoL and cost domains, only HAS addresses how to deal with QoL data, while NICE and CADTH each provide a single recommendation for PRO studies. None of the HTA guidance documents include recommendations on real-world cost. Exploring biases and statistical methods across guidance documents All guidance documents acknowledge potential biases that may arise in conducting RWE studies, including missing data and misclassification (referred to as ‘classification bias’ by HAS and EMA), but most provide only limited discussion of their definitions or likelihood. While each document names at least one statistical approach to mitigate biases and confounders, including imputation, propensity scores and quantitative bias analysis, none provide detailed instruction on their application (online supplemental tables 8 and 9). Several guidance documents also refer to reporting tools and frameworks to improve study reproducibility. For example, HARmonised Protocol Template to Enhance Reproducibility is mentioned by six agencies for developing study protocols and as a reporting tool. Variability in the tools cited partly reflects the publication dates of the guidance document, with newer documents referring to more recently developed methods (online supplemental table 10). Results of HTA interviews We conducted 13 interviews, one for each country, except Italy, where we conducted two interviews. Most experts indicated that the key principles of generating good evidence remain consistent, regardless of the data source. To validate our selection of documents, we asked experts about the availability of RWE guidance documents in their countries, or if they missed one, confirming the appropriateness of our selected documents. Two experts referenced two additional international RWE guidance documents (EUnetHTA and ISPOR), which we included in our analysis. Feasibility of RWE guidance documents and their applicability to hospital data Six experts (46%) reported that the available RWE guidance documents are feasible to implement. Four experts (31%) were unsure about their feasibility, while three experts (23%) did not respond to the question. One expert mentioned that ‘It’s always easy to make guidelines and then in practice, you know it’s much more difficult, so I think it’s work in progress.’ Regarding the applicability of these guidance documents to hospital data, five experts (38%) expressed doubt due to data heterogeneity and the lack of standardisation. Only one expert (8%) confirmed that the guidance documents cover hospital data sources, stating that ‘The principles are the same.’ Four experts (31%) did not answer the question, and three (23%) provided unclear responses. The need for hospital data-specific guidance Five experts (38%) supported the need for guidance documents specifically tailored to hospital data, emphasising the benefits of detailed instructions on data usage, standardisation across sources and addressing privacy concerns. One expert explained, ‘Because of the different nature of these sources, and given that hospital data are also collected for administrative purposes, their quality and validity require different approaches to evaluation, since the reason behind data collection is different. So I believe that some points are common across all types of sources, but probably there is a need to make a specific focus depending on the type of source.’ Three experts (23%) opposed this approach, arguing that transparency and the research question are more crucial, and that data quality criteria are generally consistent across sources. The remaining participants did not give any or inconclusive answers. Barriers and facilitators in using hospital data for HTA The interview analysis identified seven themes and five subthemes, reflecting experts’ views on the barriers and facilitators of hospital data for HTA table 3. We found 13 barriers and 15 facilitators. Barriers mainly involved data issues, including availability, access, timeliness, quality, validation and heterogeneity. Other key barriers included concerns about data generalisability, treatment and patient selection, privacy and access to individual patient data. Table 3Interview themes: barriers and facilitators for using real-world hospital data in HTAThemeSubthemeBarriersFacilitatorsData-relatedAvailabilityLimited data points (eg, HRQoL, morbidity)Collect more data pointsAccessDifficulties in data access, especially individual patient data, Slow data retrievalAccess to aggregated dataQualityMissing data, misclassification, unstructured dataEnsure reliability, improve structured data, enhance data linkageStandardisationBias due to heterogeneity, challenges identifying the study cohortHarmonise data collection, improve standardisation using CDM, mapping to OMOPValidityDifficult validation processEnsure data validationGeneralisabilityRegional variations affect treatment and patient selectionDetailed hospital site information (type, capacity)ELSI-relatedConcerns regarding privacy and access to individual patient data.Establish ethics and privacy protection criteriaOrganisational contextLack of staff and resourcesEstablish organisational networksLack of transparency in HTA processClear guidance, scientific frameworksChanges in regulations or guidelines over timeCommunication–Education regarding HTA processCollaborations, transparency, sharing codesReflection documents on the guidance missingEarly industry-regulatory communicationGovernanceChallenges with custodianship and governanceCentralised governance, clear data ownershipTechnology-relatedChallenges ensuring AI reliability and transparencyUsing artificial intelligenceAI indicates Artificial IntelligenceCDM, common data model; ELSI, Ethical Legal Social Issues; HRQoL, Health-related quality of life; HTA, Health Technology Assessment; OMOP, Observational Medical Outcomes Partnership. Interview themes: barriers and facilitators for using real-world hospital data in HTA AI indicates Artificial Intelligence CDM, common data model; ELSI, Ethical Legal Social Issues; HRQoL, Health-related quality of life; HTA, Health Technology Assessment; OMOP, Observational Medical Outcomes Partnership. Facilitators focused on improving data practices, including standardised collection, reliability and better linkage to other sources while expanding data scope for HTA. They also emphasised enhancing communication through educational materials for the scientific community and healthcare providers and addressing guidance gaps. Experts highlighted advanced technologies like the common data model and Observational Medical Outcomes Partnership (OMOP) as valuable for standardisation. Despite challenges in data quality, availability and privacy, standardising practices and leveraging advanced technologies can mitigate these issues, unlocking the full potential of hospital data for HTA. Discussion The objective of this paper is to comprehensively compare international HTA and regulatory agencies’ RWE recommendations, with a special focus on how these recommendations apply to real-world hospital data. We conducted a scoping review to identify available RWE guidance documents from international HTA and regulatory agencies, focusing on hospital data, complemented by semi-structured interviews with HTA experts. We identified 19 guidance documents related to RWE in HTA (9) and regulatory (10) contexts. Only four guidance documents (two HTA, two regulatory) explicitly have addressed hospital data, particularly in terms of EHRs.927,29 We found limited guidance on using hospital data to generate evidence for technology assessments. We observed variability in the scope, RWD definitions, terminologies and RWE applications across the guidance documents, reflecting varying levels of RWE acceptance. These findings align with existing literature highlighting a lack of harmonisation in policies, evidence requirements and RWE acceptance among HTA agencies.20 43 Inconsistencies in terminology used were also reported by studies assessing data quality terms and dimensions for the secondary use of EHRs.44 45 Collectively, these differences pose challenges for users and limit the comparability and transferability of RWE across agencies. HTA experts emphasised the need for standardised policies to address these discrepancies. Our findings highlight considerable variation in both the level of detail and focus regarding hospital data. In fact, six of the eleven agencies do not address hospital data at all, and those that do vary widely in their scope and depth. IQWiG provides population-oriented guidance with three specific recommendations, whereas FDA and ISPOR offer more comprehensive recommendations addressing data quality, completeness, interoperability and the transformation of unstructured data. NICE and CADTH mention hospital data indirectly and provide limited methodological guidance, highlighting the absence of detailed recommendations from most agencies. Notably, FDA and ISPOR guidance documents overlap in several areas, highlighting shared priorities in ensuring reliable, traceable and fit-for-purpose data. While both FDA and NICE emphasise the importance of conceptual and operational definitions for key variables, only FDA provides detailed recommendations on how to develop these definitions. This fragmented landscape reveals that, despite the central role of hospital-based data such as EHRs in generating RWE, there is no consistent or comprehensive international standard for their use in HTA. Overall, guidance documents do provide important elements, but vary in specificity and level of detail, which may lead to potential misinterpretations. These findings highlight a critical gap in existing guidance, demonstrating the need for harmonised (also in sight of the HTA Regulation and Joint Clinical Assessments), practical standards to better support the use of hospital data in HTA and improve the reliability and comparability of RWE-based assessments. Our findings show that despite their relation to hospital data in principle, limited recommendations address their unique aspects, including unstructured data. Hospital data can serve as a low-cost, real-time source of longitudinal data on large populations, which can broaden the scope of precision medicine by incorporating factors such as occupation, environment, social interactions and behaviour.46 47 HTA experts in our study acknowledged these unique characteristics, noting that unstructured data and heterogeneity require targeted approaches to validation and quality assessment, and suggested that advancing technology could support these processes. This perspective aligns with existing literature discussing the challenges and potential solutions for using hospital data, emphasising their value for improving the evidence base in clinical and health economic decision making.48 49 Although ISPOR’s SUITABILITY checklist highlights key considerations for assessing the suitability of EHR data in HTA, its approach is broad and lacks guidance on practical aspects, such as methods for validating tools used in transforming unstructured data.9 The task force also advocates for collaboration between HTA and regulatory agencies to develop and support a common set of standards related to the use of EHRs in decision-making.50 Despite these efforts, our findings show that recommendations on hospital data are distributed across guidance documents, reflecting their low priority. HTA experts also identified the lack of transparency in the HTA process as a barrier to using hospital data, and providing clear guidance could facilitate the use of these data. So far, only EHRs have been discussed in the guidance documents; however, hospital data include not only more than just EHRs but they also include resource utilisation and financial data, both of which are relevant to the HTA process. When analysing recommendations across the HTA domains of clinical effectiveness, cost and quality of life, our findings show that the guidance documents primarily focus on the clinical effectiveness domain, with broad agreement on the need for high-quality RWE and common recommendation regarding study planning, design and reporting. While the clinical-effectiveness domain remains central to HTA, the QoL and cost domains can carry comparable weight in decision-making. However, recommendations for real-world QoL are limited and general, providing varying levels of detail on the validity and relevance of the tools used. This issue was also recognised by Chassany et al, who reviewed PROs guidance documents from IQWiG, Gemeinsame Bundesausschuss (GB-A), HAS and NICE, as well as 70 oncology submissions from these agencies. They conclude that the level of detail varied, with limited guidance on analysing and reporting PROs and inconsistent evidence evaluation among these agencies.51 Therefore, more detailed and standardised guidance would enhance comparability and certainty for PROs. We observed a lack of recommendations addressing real-world cost, despite their significant role in HTA decision-making. The absence of guidance on real-world costs may reflect the complexity and heterogeneity of cost data across healthcare systems, particularly in hospital settings. Unlike clinical outcomes, cost data are often collected for administrative, budgeting or accounting purposes rather than for research or evaluation. Consequently, such data can vary widely in definitions, coding practices and granularity. Furthermore, cost structures differ considerably between countries, institutions and even departments within the same hospital, making standardisation and transferability challenging. In addition, in some countries, these data remain confidential due to health market regulations, further limiting their availability for HTA. Costs used in HTA typically reflect published costs or listed prices, which do not reflect real costs. Real-world costs, although limitedly explored in literature, could reduce uncertainty and support price negotiations by providing insights into actual costs. A survey comparing the prices of several cancer drugs across European countries highlights the variation between actual and listed prices, emphasising the need for guidance on addressing real-world cost.52 Another study examining three approaches to estimating hospital costs found considerable variation in cost outcomes based on the cost unit used, which can affect reimbursement decisions.53 Therefore, more guidance is needed on collecting, harmonising, estimating, reporting and assessing the quality, validity and representativeness of real-world costs used. The development of such guidance, ideally enriched with practical use cases, could increase transparency and consistency in the use of real-world hospital cost data for HTA, thereby supporting more accurate, context-specific and equitable decision-making. In light of this study’s findings, a clear road map is essential for guiding future guidance on hospital data use. The intended audience includes both HTA conductors and evaluators. Given the opportunities and challenges of hospital data, guidance should focus on key actionable steps. First, they should define best practices for transforming unstructured data into structured formats and ensuring text mining accuracy. Socio-technological aspects such as AI training and its implications for patient data should also be addressed. Additionally, organisational impacts, including effects on hospital operations and patient care, should be integrated. Collaboration between healthcare providers, technology developers and policymakers is essential for guidance to remain relevant. Expanding use cases of hospital data for HTA will enhance their depth. As AI becomes central to leveraging RWD, guidance should address AI’s role in processing unstructured data, integrating sources and automating tasks. They should also establish frameworks for AI validation, bias minimisation and governance. NICE has issued a position statement on AI in evidence generation.54 Addressing these considerations will improve hospital data use in technology assessment. There are certain limitations that may influence the scope of the current review. We did not include low and middle-income countries as these systems often have other policy priorities. The present country selection aimed to provide a broad international perspective and our findings might assist other countries in shaping RWD use in future. Additionally, literature databases were excluded from the search strategy, as a preliminary review showed they did not provide sufficient insight into the content of RWE guidance documents to address our research question. Given the time lag between the publication of guidance documents and their appearance in academic literature, reliance on database searches alone was considered inadequate. However, to minimise the risk of missing relevant guidance documents, country representatives were asked during semi-structured interviews to verify and supplement our selection. Additionally, some HTA and regulatory agencies distribute their recommendations across several documents, including general methodological and economic guidance documents, which were not included in this review. We assessed this by listing approaches and other guidance documents found in the analysed documents. This fragmented landscape makes it challenging for HTA professionals to stay informed about evolving RWE guidance documents, particularly given that their feasibility and impact on current practice have not yet been fully established. Further research is needed to explore this issue in more depth. This study included guidance published up to mid-2024 and incorporated expert interviews to ensure completeness of the identified frameworks. Subsequent guidance documents or updates (eg, NICE RWE guidance updates) were therefore not captured. Future research could extend this analysis to reflect recent updates, including new NICE additions on methods to support the use of unstructured data, tools for reporting studies using target trial approaches, code-list development, further guidance on external validity bias and approaches for sampling in new studies requiring primary data collection. Conclusions In conclusion, the growing number of RWE guidance documents reflects increased interest in RWE in HTA and regulatory assessments. However, these documents primarily focus on the clinical domain, often overlooking real-world safety, QoL and cost domains. Hospital data are limitedly addressed, with current guidance focusing solely on EHR data. We recommend developing guidance tailored to hospital data considering their hybrid nature (structured and unstructured). To unlock the full potential of hospital data, future guidance should provide clear instructions for developing and validating conceptual and operational definitions to identify key variables and patient cohorts, along with systematic approaches for assessing and validating text-mining tools and advanced technologies. Additionally, strategies to assess the generalisability of the data source(s) are necessary. Addressing these aspects, future guidance can elevate evidence quality and drive meaningful advancements in the use of hospital data for technology assessment. Supplementary material Acknowledgements The study was funded by the ONCOVALUE Consortium (grant agreement No 101095245). We would like to thank all members of the ONCOVALUE Consortium for their assistance in locating HTA experts for our interviews. The authors acknowledge that portions of this research were presented at the HTAi Conference, held in 2024 in Seville. The corresponding conference abstract, titled “Guidance On Using Hospital-Based Real-World Evidence In Health Technology Assessments For Oncology," was published in International Journal of Technology Assessment in Health Care and can be accessed at https://doi.org/10.1017/S0266462324002095. Footnotes Funding: This study was supported by the ONCOVALUE Consortium, funded by the European Union's Horizon Europe research and innovation programme under grant agreement No 101095245. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Prepublication history and additional supplemental material for this paper are available online. To view these files, please visit the journal online (https://doi.org/10.1136/bmjopen-2025-100304). Provenance and peer review: Not commissioned; externally peer reviewed. Patient consent for publication: All invited experts that participated in the interview consented before participating. Ethics approval: The Institutional Review Board (IRB) of the Netherlands Cancer Institute granted ethical approval to this study (IRBd23-246). Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research. Data availability free text: The guidance documents used in the analyses are available publicly on the websites listed in online supplemental table 1. Transcripts of the expert roundtable will not be shared to ensure anonymity of the participants. Data availability statement No data are available. All data relevant to the study are included in the article or uploaded as supplementary information. References Supplementary Materials Data Availability Statement No data are available. All data relevant to the study are included in the article or uploaded as supplementary information.