A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results.
Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. · 2015
Read A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. by Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. (2015) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4632358.txt.
Introduction
A 2015 literature review by Tessa Kennedy-Martin and colleagues deals with whether patients in drug trials resemble patients in everyday care. Trials run under tight rules that can exclude many typical patients. The review gathered studies comparing trial samples with real-world populations in cardiology, mental health, and oncology. It appeared in Trials.
From over five thousand abstracts, fifty-two studies qualified. Most found trial samples were highly selected and lower risk than real patients. Older people and those with several conditions were often shut out, and large shares of real patients would have failed trial entry rules. About seventy-one percent of the studies concluded openly that trial samples were not broadly representative, which limits how far trial results can be stretched.
Catalog record
| CREATOR | Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. |
|---|---|
| YEAR | 2015 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results By Tessa Kennedy-Martin, Sarah Curtis, Douglas Faries, Susan Robinson, Joseph Johnston Published in: Trials DOI: 10.1186/s13063-015-1023-4 ABSTRACT Randomized controlled trials (RCTs) are conducted under idealized and rigorously controlled conditions that may compromise their external validity. A literature review was conducted of published English language articles that reported the findings of studies assessing external validity by a comparison of the patient sample included in RCTs reporting on pharmaceutical interventions with patients from everyday clinical practice. The review focused on publications in the fields of cardiology, mental health, and oncology. A range of databases were interrogated (MEDLINE; EMBASE; Science Citation Index; Cochrane Methodology Register). Double-abstract review and data extraction were performed as per protocol specifications. Out of 5,456 de-duplicated abstracts, 52 studies met the inclusion criteria (cardiology, n = 20; mental health, n = 17; oncology, n = 15). Studies either performed an analysis of the baseline characteristics (demographic, socioeconomic, and clinical parameters) of RCT-enrolled patients compared with a real-world population, or assessed the proportion of real-world patients who would have been eligible for RCT inclusion following the application of RCT inclusion/exclusion criteria. Many of the included studies concluded that RCT samples are highly selected and have a lower risk profile than real-world populations, with the frequent exclusion of elderly patients and patients with co-morbidities. Calculation of ineligibility rates in individual studies showed that a high proportion of the general disease population was often excluded from trials. The majority of studies (n = 37 [71.2 %]) explicitly concluded that RCT samples were not broadly representative of real-world patients and that this may limit the external validity of the RCT. Authors made a number of recommendations to improve external validity. Findings from this review indicate that there is a need to improve the external validity of RCTs such that physicians treating patients in real-world settings have the appropriate evidence on which to base their clinical decisions. This goal could be achieved by trial design modification to include a more representative patient sample and by supplementing RCT evidence with data generated from observational studies. In general, a thoughtful approach to clinical evidence generation is required in which the trade-offs between internal and external validity are considered in a holistic and balanced manner. Electronic supplementary material The online version of this article (doi:10.1186/s13063-015-1023-4) contains supplementary material, which is available to authorized users. BACKGROUND Appropriately designed and executed randomized controlled trials (RCTs) represent the current gold-standard primary study design for the determination of the efficacy and safety of medical interventions [1]. Evidence from RCTs is used by healthcare providers to guide their clinical decisions and by payers and policy makers to support their recommendations for the adoption of new therapies in clinical practice [2]. Explanatory RCTs are designed to determine the efficacy of an intervention under idealized and controlled circumstances and so are conducted under rigorous conditions, including strict adherence to structured protocols, the use of restrictive inclusion and exclusion criteria, and patient randomization, that maximize their internal validity (that is to ensure they minimize the possibility of bias regarding the effect of an intervention) [3, 4]. In order for the results of such trials to be clinically useful, they must also be relevant to a definable patient population in a specific healthcare setting, a concept that is termed external validity or generalizability (note, these terms are used interchangeably [3] in this review and describe the applicability of the study results outside of the trial environment) [5–7]. As it is challenging to simultaneously optimize internal and external validity, efficacy data from traditional explanatory RCTs are often complemented by evidence from pragmatic trials (including pragmatic RCTs) or observational studies that determine the performance of an intervention under conditions more closely resembling routine clinical practice, and include more heterogeneous patient populations and less stringent treatment and delivery protocols [4]. While some pragmatic trials have good internal validity and some observational studies may lack external validity, generally explanatory RCTs tend to maximize internal validity at the expense of external validity, while studies conducted in a setting more closely resembling real-world practice may do the opposite. As such, evidence from all these sources can be complementary in understanding the effect of an intervention and furthering clinical research [8]. In recent years, the need to better understand the external validity of RCT results has been identified across numerous therapeutic areas [9–13]. However, a comprehensive literature review of studies that have assessed the representativeness of RCT populations has not been undertaken in recent years (note, the term representativeness has been used throughout this review to describe the similarities between RCT samples and real-world populations). To examine this issue, we conducted a literature review of studies that have attempted to evaluate external validity in one of two ways: (i) by comparing the clinical characteristics of an RCT sample with those of everyday clinical practice patients, or (ii) by assessing what proportion of a real-world population would satisfy the criteria for RCT inclusion. In the context of the current review, real-world populations are defined as those patients encountered in routine clinical practice settings (for example, patients included in observational cohorts or patients identified from medical chart review, registries, or insurance databases). The primary objective of the review was to assess the extent to which RCT samples are representative of real-world populations (which may or may not affect the external validity of the trial findings). Other objectives were to identify key issues that may impact the external validity of trial findings (with reference to included studies) and also to outline recommendations from the identified studies for improving external validity. The present review was limited to RCTs in oncology, mental health, and cardiology as, when the review was undertaken, these were identified as the main therapeutic areas in which RCT and real-world populations had been compared. It should be noted that the focus of the current review was explanatory and not pragmatic RCTs. REVIEW Methods The methodological framework of this literature review was employed to examine the extent, range, and nature of research activity regarding the representativeness of RCT patient samples and the implications of this to the external validity of the findings. The review involved a five-stage process [14]: identification of the research question; identification of studies relevant to the research question; selection of studies to include in the review; charting of information and data within the included studies; collating, summarizing, and reporting results of the review. A search protocol was written that outlined the objectives, search methods, and the process for study selection and data extraction. Information sources, search approach, and strategy Searches were run in MEDLINE and MEDLINE In-Process, EMBASE, Science Citation Index, and the Cochrane Methodology Register and were supplemented with reference checking. When combined with citation searching, these sources presented a reasonable basis for a targeted search of the published literature. The searches were run on 30 September 2013 and included published studies conducted from 2003 to 2013 in order to reflect contemporary clinical trial practice. A base-case search strategy was created in the Ovid MEDLINE interface, and once finalized, was adapted to meet the syntax of the other databases. See