Why Most Clinical Research Is Not Useful.
Ioannidis JP. · 2016
Read Why Most Clinical Research Is Not Useful. by Ioannidis JP. (2016) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc4915619.txt.
Introduction
John P. A. Ioannidis argued in PLoS Medicine in 2016 that most clinical research is not useful, and that reform is overdue. Blue-sky work cannot be judged by direct impact, but clinical studies should aim to change health outcomes. Eight traits mark useful work: a real problem base, fit with prior proof, fresh information, practical design, patient focus, value for money, feasibility, and openness.
Many studies, even in top journals, fail several traits at once: tiny samples, questions nobody needs, or designs no clinic could copy. Ioannidis lists fixes for funders, editors, and teams. The essay is a yardstick readers can lay against any trial report to judge whether the work can matter.
Catalog record
| CREATOR | Ioannidis JP. |
|---|---|
| YEAR | 2016 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Why Most Clinical Research Is Not Useful By John P. A. Ioannidis Published in: PLoS Medicine DOI: 10.1371/journal.pmed.1002049 ABSTRACT John Ioannidis argues that problem base, context placement, information gain, pragmatism, patient centeredness, value for money, feasibility, and transparency define useful clinical research. He suggests most clinical research is not useful and reform is overdue. Summary PointsBlue-sky research cannot be easily judged on the basis of practical impact, but clinical research is different and should be useful. It should make a difference for health and disease outcomes or should be undertaken with that as a realistic prospect.Many of the features that make clinical research useful can be identified, including those relating to problem base, context placement, information gain, pragmatism, patient centeredness, value for money, feasibility, and transparency.Many studies, even in the major general medical journals, do not satisfy these features, and very few studies satisfy most or all of them. Most clinical research therefore fails to be useful not because of its findings but because of its design.The forces driving the production and dissemination of nonuseful clinical research are largely identifiable and modifiable.Reform is needed. Altering our approach could easily produce more clinical research that is useful, at the same or even at a massively reduced cost. Practicing doctors and other health care professionals will be familiar with how little of what they find in medical journals is useful. The term “clinical research” is meant to cover all types of investigation that address questions on the treatment, prevention, diagnosis/screening, or prognosis of disease or enhancement and maintenance of health. Experimental intervention studies (clinical trials) are the major design intended to answer such questions, but observational studies may also offer relevant evidence. “Useful clinical research” means that it can lead to a favorable change in decision making (when changes in benefits, harms, cost, and any other impact are considered) either by itself or when integrated with other studies and evidence in systematic reviews, meta-analyses, decision analyses, and guidelines. There are many millions of papers of clinical research—approximately 1 million papers from clinical trials have been published to date, along with tens of thousands of systematic reviews—but most of them are not useful. Waste across medical research (clinical or other types) has been estimated as consuming 85% of the billions spent each year [1]. I have previously written about why most published research is false [2] and how to make more of it true [3]. In order to be useful, clinical research should be true, but this is not sufficient. Here I describe the key features of useful clinical research (Table 1) and the current state of affairs and suggest future prospects for improvement. [Table: Features to consider in appraising whether clinical research is useful.] Making speculative, blue-sky research more productive represents a partly intractable problem, given the unpredictability of such research, but significantly improving clinical research—and developing tools for assessing its utility or lack thereof—appears conceptually more straightforward. FEATURES OF CLINICALLY USEFUL RESEARCH Problem Base There is higher utility in solving problems with higher disease burdens. However, context is important. Solving problems with low prevalence but grave consequences for affected patients is valuable, and broadly applicable useful research may stem from studying rare conditions if the knowledge is also relevant to common conditions (e.g., discovering the importance of the proprotein convertase subtilisin-kexin type 9 [PCSK9] pathway in familial hypercholesterolemia may help develop treatments for many other patients with cardiovascular disease). Furthermore, for explosive epidemics (e.g., Ebola), one should also consider the potential burden if the epidemic gets out of control. Conversely, clinical research confers actual disutility when disease mongering [4] creates a fictitious perception of disease burden among healthy people. In such circumstances, treated people, by definition, cannot benefit, because there is no real disease to treat. Data show only weak or modest correlations between the amount of research done and the burden of various diseases [5,6]. Moreover, disease mongering affects multiple medical specialties [4,7,8]. Context Placement and Information Gain Useful clinical research procures a clinically relevant information gain [9]: it adds to what we already know. This means that, first, we need to be aware of what we already know so that new information can be placed in context [10]. Second, studies should be designed to provide sufficiently large amounts of evidence to ensure patients, clinicians, and decision makers can be confident about the magnitude and specifics of benefits and harms, and these studies should be judged based on clinical impact and their ability to change practice. Ideally, studies that are launched should be clinically useful regardless of their eventual results. If the findings of a study are expected to be clinically useful only if a particular result is obtained, there may be a pressure to either obtain that result or interpret the data as if the desired result has been obtained. Most new research is not preceded or accompanied by systematic reviews [10,11]. Interventions are often compared to placebos or normal care, despite effective interventions having previously been demonstrated. Sample-size calculations almost always see each trial in isolation, ignoring other studies. Across PubMed, the median sample size for published randomized trials in 2006 was 36 per arm [12]. Nonvalidated surrogate outcomes lacking clinical insight [13] and composite outcomes that combine outcomes of very different clinical portent [14] are often utilized so that authors can claim that clinical studies are well powered. The value of “negative” results is rarely discussed when clinical studies are being designed. Pragmatism Research inferences should be applicable to real-life circumstances. When the context of clinical research studies deviates from typical real-life circumstances, the question critical readers should ask is, to what extent do these differences invalidate the main conclusions of the study? A common misconception is that a trial population should be fully representative of the general population of all patients (for treatment) or the entire community (for prevention) to be generalizable. Randomized trials depend on consent; thus, no trial is a perfect random sample of the general population. However, treatment effects may be similar in nonparticipants, and capturing real-life circumstances is possible, regardless of the representativeness of the study sample, by utilizing pragmatic study designs. Pragmatism has long been advocated in clinical research [15], but it is rare. Only nine industry-funded pragmatic comparative drug effectiveness trials were published between 1996 and 2010 according to a systematic review of the literature [16], while thousands of efficacy trials have been published that explore optimization of testing circumstances. Studying treatment effects under idealized clinical trial conditions is attractive, but questions then remain over the generalizability of the findings to real-life circumstances. Observational studies (performed in the thousands) are often precariously interpreted as able to answer questions about causal treatment effects [17]. The use of routinely collected data is typically touted as being more representative of real life, but this is often not true. Most of the widely used observational studies deal with peculiar populations (e.g., nurses, physicians, or workers) and/or peculiar circumstances (e.g., patients managed in specialized health care systems or covered by specific insurance or fitting criteria for inclusion in a registry). Eventually, observational studies often substantially overestimate treatment effects [18,19]. Patient Centeredness Useful research is patient centered [20]. It is done to benefit patients or to