Appraisal of umbrella reviews on vaccines Authors: Rosa Katia Bellomo, John P A Ioannidis Published: 2024 ABSTRACT AbstractUmbrella reviews are systematic reviews of all systematic reviews on a given question and can provide a higher-level view of current available evidence on broad research topics. Given the importance and possible consequences of the insights derived from umbrella reviews, it is fundamental to ensure accurate, comprehensive reporting and methodological rigor in these papers. Here, we present the potential and limitations of these evolving evidence synthesis tools by looking at umbrella reviews published on vaccine research. Vaccines are essential tools in modern medicine for preventing disease and promoting public health. The scientific literature on vaccines grows rapidly as researchers work to improve existing vaccines and develop new ones. More than half a million scientific papers on vaccines have already been published, including several thousands of systematic reviews that aim to appraise and summarize this growing literature. However, as the published systematic reviews are only able to cover a small proportion of the literature it is important to also provide higher-level evidence syntheses. and discuss further improvements that could be made to integrate evidence on vaccines.Subject terms: Health care, Medical research Introduction Systematic reviews aim to identify, appraise, and synthesize the available evidence on a specific research question, in a transparent and rigorous manner according to explicit criteria. They are considered to be the highest level of evidence in evidence-based medicine and practice and are often used to inform clinical guidelines and policy decisions for individuals and populations. They have become increasingly popular in recent years1. Across all PubMed, systematic reviews account for approximately 3% of recently published papers (53,208 of the 1,872,812 papers published in 2021). Searching for “systematic review” on PubMed, shows that 252,031 articles classified under this type were published from 2000 to 2022, with 4246 using the search strategy “systematic review” AND “vaccine”. Nevertheless, despite their undeniable scientific value, systematic reviews hold a series of controversies and limitations2–5.Systematic reviews are intended to be objective and unbiased, but there is the potential for bias to be introduced at various stages of the review process. For example, the inclusion and exclusion criteria for studies may be subjective, and the quality assessment of the studies may be influenced by the reviewers’ beliefs or assumptions6,7. Furthermore, systematic reviews are often based on published studies, which may lead to publication bias, as positive or statistically significant results are more likely to be published than negative or non-significant results. This can lead to an overestimation of the effect size or strength of studied associations. In addition, data on adverse events can often be more fragmented and given more cursory treatment than data on efficacy or effectiveness8,9. Systematic reviews are not designed to be able to remedy the fragmentation and problems of data availability. Most systematic reviews focus on relatively narrow, circumscribed topics and specific research questions. Thus, they can offer a fragmented and sometimes even misleading picture, if other aspects of the relevant research question(s) are not also addressed10. They may also be based on a limited number of studies that are not representative of the wider population. This can lead to findings that are not generalizable. Often the included studies are very heterogenous, so that drawing valid conclusions might be a challenging task both for the authors and the researchers approaching the topic of interest9. Moreover, as this type of evidence synthesis becomes more popular, it is increasingly common to have many systematic reviews that touch on similar or overlapping research questions and topics10. This can create confusion, particularly if results on similar or overlapping questions are seemingly contradictory and discrepant11–13. Even in the absence of contradictory reviews, it is important to see the broader picture of the evidence, which may have been summarized in many different systematic reviews, each covering one small aspect of the field of interest. Umbrella reviews have emerged to provide the required higher level of evidence synthesis. In this Perspective we describe the variety of methods that have been used to undertake Umbrella reviews, using publications from the vaccination area to demonstrate the diversity of approaches adopted and the issues that result from this. Scientific literature and systematic reviews on vaccines Vaccines prevent illness, promoting overall health. They have been responsible for the eradication of smallpox and the control of a broad number of infectious diseases, such as polio and measles. Today, there is a wide range of vaccines available for a variety of infectious diseases, and the scientific literature on vaccines continues to grow as researchers work to improve existing vaccines and develop new ones. This burgeoning scientific literature involves a broad spectrum of research, including (but not limited to) clinical trials that evaluate the safety and effectiveness of vaccines in humans, laboratory studies that investigate the mechanisms of how vaccines work, and observational studies that examine the impact of vaccines on population health. There are also many publications on the economic and social impact of vaccines, including studies that examine the cost-effectiveness of vaccination programs and the impact of vaccines on public health. Overall, the scientific literature on vaccines provides considerable information on the benefits and potential risks of vaccination and is an essential resource for policymakers, healthcare providers, and the general public. However, given the complexity and occasional controversies in the field, it can be difficult to remain appraised of, and interpret evidence from, this growing number of extensive scientific publications in a balanced, systematic way. By the beginning of 2023, more than half a million scientific papers on vaccines have been published. A search of PubMed using the search string (vaccin*) OR (immuniz*) on March 12th, 2023 yielded 599,421 results, the vast majority of which are about vaccination and related themes. The growth of literature on vaccines over time is faster than the growth of the overall PubMed-indexed literature, with a substantial increase seen in number of published papers and systematic reviews from 2020 and during the COVID-19 pandemic (Figs. 1, 2)1. Number of articles and systematic reviews published between 2000 and 2022. Data is from PubMed. x axis: years, y axis: n. of publications. A Evolution of scientific literature over time. Blue line: all scientific articles, orange line: scientific articles on vaccines. B Overall SR and Vaccine SR production. Blue line: systematic reviews, orange line: systematic reviews on vaccines. Umbrella reviews or overviews of reviews published over time in PubMed. x axis: year of publication, y axis: n. of umbrella reviews. Umbrella reviews Umbrella reviews or overviews of reviews, combine evidence deriving from multiple systematic reviews on a broader topic or field of scientific inquiry. Searching PubMed with the string “umbrella review*“ OR “overview of review*“ OR “overview of systematic reviews” or “review of reviews”” obtained 2985 results published before April 2023. The first article that mentioned this new kind of literature was published in 199814, with the article type only increasing in number about 10 years later (Fig. 2). An umbrella review is a tool that can provide a bird’s eye view of current available evidence on broad research topics. By definition, it is a systematic collection and assessment of multiple systematic reviews and meta-analysis (SRMAs) on a wide research topic, developed to deal with the ever-increasing number of this type of studies in biomedical literature. The goal of an umbrella review is to assess the strength of the available evidence and summarize what is known. In this way, they can also help in setting priorities for future research. While other review types, such as scoping reviews15,16, aim to broadly map the landscape of existing literature on a topic, umbrella reviews can delve deeper and wider by rigorously synthesizing evidence from multiple existing systematic reviews, each of them covering a single specific topic, but cumulatively covering a much broader space. There is a confusing nomenclature applied to these types of articles, including “scoping review”, “mapping review”, and “evidence and gap map”17. Some of these projects may share characteristics of umbrella reviews. Nevertheless, in principle, a typical scoping review starts from searching the primary research literature. Conversely, an umbrella review builds on the availability of multiple systematic reviews that have already been conducted. The comprehensive and meticulous approach adopted whilst undertaking an umbrella review makes them particularly suitable for informing clinical guidelines and decisions, as they may provide a clearer and more robust overview of the best available evidence in a whole field. Umbrella reviews are particularly useful when there is a large and rapidly growing body of literature on a topic. Umbrella reviews typically follow a rigorous methodology to ensure the validity and reliability of the findings. This includes a thorough search of the literature, the use of explicit inclusion and exclusion criteria, the assessment of the risk of bias in the included studies, and the use of appropriate statistical methods to synthesize the findings. Given the importance and possible consequences of the results deriving from umbrella reviews, it is fundamental that clinicians, researchers, decision-makers and other readers should search for specific characteristics indicating methodological rigor and meticulous reporting standards in these increasingly influential papers. Scientific literature and systematic reviews on vaccines In the context of vaccine research, umbrella reviews can be a valuable tool both for summarizing the existing evidence on different vaccines, and for assessing the results of the existing interventions implemented by public health institutions and policymakers, henceforth identifying any potential gaps in the literature that need to be further explored. Umbrella reviews performed to-date on vaccines We evaluated all the umbrella reviews and overviews of systematic reviews published about vaccines and immunization-related topics to enable us to better understand the characteristics of this article type. We searched PubMed using the query: (vaccin* OR immunizat*) AND (umbrella [ti] OR review of reviews [ti] OR overview of reviews [ti] OR overview of systematic reviews [ti]). Searching until April 18 2023 found 49 publications. We screened the articles and included those defined as an umbrella review or review of reviews or overview of reviews or overview of systematic reviews. We included reviews that were explicitly on vaccines/vaccinations/immunizations or included an identifiable subset of articles that focused on vaccines/vaccinations/immunizations. Therefore, systematic reviews with a wider focus than that of vaccination (e.g., preventive measures in general) but containing some vaccination reviews were included for consideration of their eligible component. Of the 49 retrieved items, 24 were excluded as not eligible (16 were not on vaccinations, n were protocols,1 was a preprint, 1 was an editorial, and3 were not umbrella reviews). Twenty-five papers were included in our final assessment. All were in English and published between 2005 and 2022. Out of 25, 20 were umbrella reviews conducted specifically on vaccines, while 5 had a wider focus on preventive measures that partially included vaccination strategies as well. We reviewed our collection of 25 eligible papers assessing and describing how the authors followed the key steps in writing umbrella reviews, to improve our understanding of progress in this specific field. The umbrella reviews included in our assessment investigated a broad range of vaccination topics. Six focused on influenza vaccines, five on HPV, two on childhood vaccines, three on COVID-19 vaccines, three on health inequalities relating to vaccinations, two on varicella vaccines, two on vaccinations during pregnancy and one on tuberculosis (Table 1). A total of 959 vaccination-related systematic reviews were included in these 25 umbrella reviews. 24 of the 25 umbrella reviews specified the number of original studies that were included in the systematic reviews: the sum was 20,383 original studies, although it is possible that some studies overlapped across reviews. Table 1Included umbrella reviews on vaccinesAUTHORN. of SR includedN. of original articlesSpecific topicProtocol RegistrationReporting methodologyMethodology tool usedvan der Wouden et al., 359121Preventing influenzaaNoNoLassi et al., 45148N/AInterventions to improve neonatal healthaYesYesCochrane guidelinesBosch-Capblanch et al., 7154N/AResearch on equity attributes in vaccinationsNoYesPROGRESS-Plus/PRISMADumit et al., 466167Use of eHealth with immunizationsNoNoFoss et al., 4212406Communication around HPV vaccinations for adolescents in low- and middle-income countriesYesYesPRISMADomnich et al., 5528758Influenza vaccine induced immune responseNoYesNo tool specifiedCollin et al., 561871046Interventions for reducing TB incidenceaYesYesNo tool specifiedVilla et al., 3630423Safety, efficacy and effectiveness of HPV vaccinesYesYesJBIRodriguez et al., 4716540HPV vaccinations among adolescents in the USANoYesPRISMADemurtas et al., 37656Efficacy and safety of influenza vaccination in older peopleYesYesPRISMARodrigues et al., 38448Impact of influenza vaccination in patients with cardiovascular diseaseNoYesPRISMAMacias Saint-Gerons et al., 5717274Adverse events associated with recommended vaccines during pregnancyYesYesPRISMABesnier et al., 2960453Public health interventions to reduce mobidity, mortality and health inequalities from infectious diseases in low- and middle-income countriesaYesYesPRISMAJuvet et al., 3916202Safety of influenza A H1N1 pdm09 vaccinesNoYesPRISMAKaufman et al., 30301018Parent-level barriers to uptake of childhood vaccinationYesYesJBI + PRISMAOkoli et al., 63481711Characteristics and methodological standards across systematic reviews on influenza vaccinesYesYesCochrane guidelinesBennett et al., 3110110Interventions to improve HPV vaccine uptakeYesYesJBI + PRISMATran et al., 4034563Cost Effectiveness of Human Papillomavirus VaccinesYesYesPRISMASun et al., 7218492COVID-19 vaccine response and safety in patients with cancerYesYesPRISMAAhern et al., 3320221Clinical efficacy and effectiveness of alternative varicella vaccination strategiesYesYesPRISMAKafadar et al., 48311372Determinants for COVID-19 vaccine hesitancy in the general populationYesYesPRISMAAhern et al., 3417119Safety of varicella vaccination strategiesYesYesPRISMANichol et al., 4919570Barriers and facilitators of vaccine hesitancy during pregnancy and in mothers of infantsYesYesPRISMA, JBI, Cochrane guidelines, WHO model of determinants of vaccine hesitancyAyorinde et al., 411089247Health inequalities in infectious diseasesaYesNoMcCready et al., 3231462Covid-19 vaccine hesitancy among healthcare workersYesYesJBI + PRISMASR systematic review, JBI Joanna Briggs Institute.aArticles that cover wide topics, including a subset of systematic reviews on vaccinations. Included umbrella reviews on vaccines SR systematic review, JBI Joanna Briggs Institute. aArticles that cover wide topics, including a subset of systematic reviews on vaccinations. Key steps in umbrella reviews Details of the process of conducting and reporting an umbrella review are reviewed by Belbasis et al.18 and available as reporting guidance19. In summary, carrying out an umbrella review is a complex process with four key steps: systematic literature search and study selection; data extraction; statistical analysis and grading of evidence and interpretation of findings. The research question of interest needs to be clearly defined at the start to allow explicit eligibility criteria to be defined. Then a search algorithm must be constructed to identify all possible systematic reviews that fall within the defined research area and a comprehensive search of relevant databases and sources conducted to identify all relevant systematic reviews or meta-analyses on the topic of interest. An independent double screening of eligible studies should determine the included papers. In this phase, it is important to set criteria to apply when multiple systematic reviews on the same topic show partial or complete overlap, to decide which systematic reviews to include in the umbrella review. Researchers can choose the most recent systematic review or the systematic review with the largest number or studies or base their choice on the quality of the selected papers. The quality of the included systematic reviews must be assessed using predefined tools or checklists. One may also try to consider the risk of bias (ROB) of the original studies included by the systematic reviews. Alternatively, such an assessment may have been done already by the systematic reviews, and in this scenario the collected information needs to be captured and compiled across all eligible systematic reviews. Eligibility criteria and how they fit with the research question(s) of interest should be explicitly stated. Relevant data from the included studies need to be extracted and tabulated, synthesizing the findings of the included studies using appropriate methods such as meta-analysis, narrative synthesis, or a combination of both, with a description of the reasons for this choice. The statistical analyses to estimate all the relevant features about heterogeneity between the studies included in meta-analyses within the systematic reviews should be repeated using consistent methods across all systematic reviews and meta-analyses. This may be difficult to perform if the data are not presented in sufficient detail in some studies. In this case, the meta-analyses’ results can be recorded. If meta-analyses can be re-performed in a standard way, then summary estimates, 95% confidence intervals, 95% prediction intervals can be calculated and assessed in a consistent manner. Moreover, statistical tests of bias, such as asymmetry tests for small study effects (evaluating whether small studies give different results than larger studies)20,21 and excess significance tests (evaluating whether there are spuriously too many studies with statistically significant results)22,23 can be performed. The overall strength of the evidence can also be graded according to pre-specified sets of comprehensive criteria, e.g. GRADE (Grading of Recommendations, Assessment, Development and Evaluation)24,25 for questions about interventions, while for epidemiological associations an assessment can be made by considering features such as the amount of evidence, the significance level, the extent of heterogeneity between studies and hints for potential bias. Lastly, the findings of the umbrella review need to be interpreted in the context of other relevant existing literature to draw comprehensive conclusions. Methodological analysis of 25 umbrella reviews on vaccines We evaluated the methodological rigor of 25 umbrella reviews on vaccines, defining this as the strictness and consistency with which research methods were applied and adhered to recognized standards. Specifically, in our analysis, a variety of methodological features were assessed, including the presence of protocol registration, clear reporting methodology and analysis, search algorithm, comprehensive literature search, double screening, inclusion and exclusion criteria, PRISMA flowchart26, list and the characteristic of the included studies, list of the excluded studies, overlap and redundancy assessment, quality assessment of the included studies, tabulation of relevant extracted data, grading of evidence and COI and funding disclosures. These methodological features contribute to how robust and reproducible an umbrella review is, and they offer some protection from diverse biases, thereby adding credibility to the conclusions drawn. The methodological rigor of the 25 umbrella reviews included in our assessment varied substantially. While some of the aforementioned steps were followed consistently by almost all umbrella reviews, some others were rarely fulfilled (Fig. 3). Fulfilment of methodology key steps in vaccination Umbrella Reviews. x axis: % of articles, y axis: key methodological steps. Blue bar: key methodological step fulfilled, Pink bar: key methodological step not fulfilled. All of the selected papers included either the terms “umbrella review” or “systematic reviews of systematic reviews” or “overview of systematic reviews” in their titles and overall had a clear research question. Eighteen out of twenty-five studies provided a registration number (most commonly PROSPERO) and all but three stated that they used a clear reporting guideline tool, which was most commonly PRISMA (seventeen papers)26 or a Joanna Briggs guidance checklist (five papers)27 or a Cochrane tool (three papers)28 (Table 1). There was no search algorithm in just one paper, and in all but two papers a comprehensive database search was performed (searching through at least two literature databases). Article selection screening in duplicate was carried out in seventeen of the umbrella reviews and a PRISMA flowchart was presented in twenty (Supplementary Data 1). Only six papers clearly indicated the total number of original primary studies included, either in the PRISMA flowchart29, in dedicated figures30, in the findings section of the article31 or during the overlap assessment32–34. Twenty-three umbrella reviews provided a list of the included systematic reviews; it was missing from two papers, where the supplementary material in which the list was supposed to be provided was reported in the paper but not uploaded online. However, the characteristics of the included systematic reviews weren’t always reported in an adequately detailed way. The excluded list with the reasons for exclusion, conversely, was only produced by twelve umbrella reviews (Supplementary Data 1). In our sample, a total of 25 studies explicitly defined their inclusion and exclusion criteria. However, one study did not furnish satisfactory criteria (van der Wouden, 2005)35. Among the 25 studies that provided criteria, there was noticeable heterogeneity in the clarity of their presentation. As a case in point, only 12 of these studies offered a clear tabulation of the reasons for inclusion29,30,32–34,36–42 (Supplementary Data 1). The quality of the systematic reviews included in the umbrella reviews was appraised in 24 of the 25. The most commonly used appraisal method was AMSTAR-243 (in 14 papers). Four used AMSTAR44, including Lassi et al.45–47. Four used the standardized JBI critical appraisal tool27, including Bennett et al.31,32[,48,49. Two articles, Rodrigues et al.38 and Kaufman et al.30, used ROBIS50. One article by van der Wouden et al.35 did not use any tool for assessing the quality of the included studies. The risk of bias within the included systematic reviews was addressed in almost all umbrella reviews (21 out of 25), but a specific report of the tools used within the included systematic reviews to appraise the primary studies was provided only in nine of them (Supplementary Data 2). Twenty-one out of twenty-five umbrella reviews elaborated their findings narratively, but only nine explicitly stated the reason for such a choice, which most of the times was due to critical methodological flaws and heterogeneity across the included systematic reviews (Supplementary Data 2). Reporting summary Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article. Overlap assessment and redundancy One of the most common methodological flaws in the overviews was the lack of overlap assessment, which was very heterogeneous among the 25 umbrella reviews. In this context, “overlap” refers to the inclusion of overlapping primary studies in multiple systematic reviews. Such overlaps arise when various reviews incorporate some of the same original research articles into their analyses, often because they address closely related research questions or rely on cornerstone references that are widely used. While overlap is not inherently problematic and may be unavoidable, its extent should be evaluated to allow evidence to be contextualized and interpreted appropriately. Only twelve of the umbrella reviews mentioned the potential overlap issue in their methodology or discussion sections. Ten of them followed a rigorous methodology in assessing overlap of primary studies, four pointed out narratively some overlaps while describing the findings of their research, some others (n = 11) did not mention this issue at all. When overlap was assessed, it was a reason for exclusion of systematic reviews in only 3 (12%) umbrellas; the most common instrument to measure it was the Corrected Covered Area (CCA, a citation matrix of all primary studies included for each systematic review converted into an index through a mathematical formula)51. This approach was used in eight (32%) of the examined umbrella reviews. However, apart from reporting the overlap score, only three authors (12%) used the grade of overlap as an exclusion criterion. Conversely, authors of umbrella reviews that chose to elaborate the evidence narratively (5, 20%) addressed the issue of overlap only in the discussion section, instead of using it as an exclusion criterion. CCA was the only approach derived from previous literature. In some umbrella reviews the authors addressed the issue by autonomously trying to identify overlapping studies based on time of publication or through specific matrix extraction methods. However, apart from reporting the overlap score, only three authors used the grade of overlap as an exclusion criterion. Authors of umbrella reviews that chose to elaborate the evidence narratively addressed the issue of overlap only in the discussion section, instead of using it as an exclusion criterion29. If correctly assessed, especially in fields such as public health and in narrative synthesis, overlap can be a benefit and not necessarily a bias52, because even if considering the same primary studies, different aspects of interventions and outcomes might be assessed, drawing dissimilar conclusions that can highlight distinct aspects worth of further future investigation, if this issue is acknowledged and addressed adequately. Only two umbrella reviews29,32 provided a detailed description of the methods used to assess overlap. Specifically, in the supplementary materials, they shared the citation matrix they used (developed by Thomson et al and51), reporting the number of individual studies actually covered in the results, to reflect the non-overlapping size of the evidence base and explicitly stating why the narrative nature of their review did not require use of overlap as an exclusion criterion (Supplementary Data 3). Another critical aspect is the assessment of redundancy according to PICO elements. Redundancy in this context refers to situations where a systematic review either replicates the research questions of existing reviews without adding new information or includes a substantial overlap of primary studies. Factors such as the scope, temporal relevance, and methodological rigor also play roles in determining the extent of redundancy. Recognizing and addressing redundancy is essential to avoid double counting in umbrella reviews. In our sample, it became evident that the predominant focus of authors was on the aspect of overlap when considering redundancy in systematic reviews. This emphasis on overlap, while noteworthy, overshadowed other crucial aspects of redundancy. Only two papers37,38 assessed redundancy among the included systematic reviews during the selection process of included papers. Specifically, the authors from Rodrigues et al.37 described and tabulated their process of selection of reviews among those evaluating the same outcome. In the supplementary materials, they provided the reasons why they considered the articles they decided to include the most representative for the outcome of interest, choosing them according to the time of publication (selecting the most recent papers), favoring reviews that focused on participants more concordant with the population of interest, that included RCTs or in which a pooled analysis had been performed (Supplementary Data 3). In Demurtas 202036, the authors performed meta-analyses to synthesize their evidence and when selecting the papers to include, they chose the largest in terms of studies. They also estimated prediction interval, in order to account for between-study effects and assess the certainty of the association if a new study addressed that same association (Ioannidis, 2009)53. Grading of the evidence Assessing the strength and rigor of evidence is pivotal in evidence synthesis, and grading is a methodological tool that serves this purpose. Grading provides a structured approach to evaluate the quality and relevance of the evidence, guiding its interpretation and its application in decision-making. In the context of umbrella reviews, it operates on two distinct levels: the grading of evidence within the systematic reviews and the grading of the umbrella reviews themselves. This layered approach introduces nuances, as the former assesses individual studies while the latter synthesizes and evaluates a collective body of evidence. From our sample, only nine included evidence grading, reflecting the challenges of this dual-layered system. The methodologies and tools used for grading were diverse and reflected the varied nature of the umbrella reviews. Lassi et al.45 used the GRADE (Grading of Recommendations, Assessment, Development and Evaluation)25 approach to assess their included studies’ evidence strength. Foss et al.42 structured their own evidence applicability using an approach adapted from another Cochrane (Effective Practice and Organisation of Care (EPOC) overview on a similar topic54. Domnich et al.55 applied Cumulative Evidence Synthesis (CES) to categorize evidence based on predefined summary effect sizes. Collin et al.56 based their grading on the evidence from included studies, classifying the overall level of evidence into categories such as “sufficient” and “insufficient.” Demurtas et al.37 performed a meta-analysis and then used GRADE for the evidence assessment. Both Rodrigues et al.38 and Ahern et al.34 extracted GRADE scores from their included systematic reviews without conducting their own grading. Macias Saint-Gerons et al.57 categorized conclusions based on available evidence robustness and synthesized findings with narrative summaries. Bennett et al.31 noted that only one included review utilized GRADE, leading to challenges in making an overall assessment for each outcome. As previously pointed out, most of the selected papers summarized the evidence narratively, reflecting the narrative and heterogeneous nature of the evidence. This makes grading the strength of evidence challenging, which was assessed only in nine umbrella reviews across our sample. Some of the authors of the papers we assessed only graded the evidence of the studies included in their systematic reviews, but most often the final evidence was not strong enough or it appeared too heterogenous for a grading of the umbrella review findings (Supplementary Data 3). Disclosures in umbrella reviews Funding disclosure sections were missing from three umbrella review papers, ten reported no funding and twelve declared the source of funding. Most of the reviews were funded by government institutions or universities and only one received partial funding from industry partners31 (Supplementary Data 4). Two umbrella review papers did not provide a conflict-of-interest section. Two had a conflict-of-interest disclosure, while all the others declared none (Supplementary Data 4). Ten papers had a data sharing statement but only two provided a code availability statement (Supplementary Data 4). Our findings underscore varying degrees of transparency and disclosure practices across the umbrella reviews. It is not possible to know whether funding information was complete. Similarly, the lack of any items mentioned in most conflict-of-interest disclosures may mean that no conflicts were involved or that conflicts were not revealed. This further emphasizes the need for a standardized approach to maintain trust and transparency. Data sharing and code availability are also paramount for fostering transparency and replicability in research58. Transparent data and code practices can bolster confidence in the findings and facilitate further research59,60. Evolution of reporting and methodological rigor over time Whilst we did find a diversity of approach, and aspects that should be covered in all Umbrella reviews missing from many of them, overall, we observed a potential trend for some improvements over time of some features of the published umbrella reviews, especially when it came to the reporting methodology and formal organization of the manuscripts. This may represent the accumulation of more experience undertaking these studies and it is also concomitant with the introduction and wider use of specific reporting and grading tools and scales, such as PRISMA, AMSTAR or PRIOR19,26,43,44. It is unclear whether there were stricter requirements for publication from journals or umbrella review authors themselves opted for better documentation. Disadvantages and limitations of umbrella reviews Our specific analysis highlights specific examples of the limitations of umbrella reviews. The validity of the included systematic reviews depends on the conduct of such reviews, some of which may be of poor quality10, impacting the conclusions that can be drawn from an umbrella review. Further, as this is a meta-meta- synthesis, the potential for bias in the primary research included in systematic reviews is rarely accounted for with bias adjusted meta-analysis, sensitivity analysis or appropriate considerations given in the conclusions of the systematic reviews. Therefore, there are more levels at which bias can occur and thus the conclusions of an umbrella review can become unreliable. Umbrella reviews also do not include information for epidemiological associations or interventions that have not been already examined in the eligible systematic reviews thus there is potential for the exclusion of important exposures, interventions, outcomes, and other factors that might be an important part of the investigation of the umbrella review. This also includes the unavoidable exclusion of some recent research if systematic reviews are not very recent themselves. Moreover, some relevant original studies might not be included because they elude the inclusion/exclusion criteria of the systematic reviews identified for the overview. The methodological standards across systematic reviews are very heterogeneous. Despite the availability of numerous methodological rigor and reporting quality assessment tools, as detailed by Kolaski et al. 61, determining which studies to include in the overview and how to grade the provided evidence can be challenging. This can sometimes come at the cost of losing important primary data. Lastly, as our work showed clearly, often it is difficult to assess the overlapping data and redundancy across the primary studies of the systematic reviews and to determine the inclusion criteria when this happens. There are several efforts underway to overcome these limitations, and guidelines addressing these issues have already been published. The most relevant ones were provided by the Joanna Brigs Institute27,62 that produced a Critical Appraisal Checklist for Systematic Reviews and Research Syntheses which was used in four of the 25 umbrella reviews that we assessed here30–32,36; and the Cochrane Handbook of Systematic Reviews of Interventions guidelines28 that was adopted by two of the umbrella reviews on vaccines45,63. The latest reporting guidelines for umbrella reviews were introduced in 2022 by a group of scientists who designed the PRIOR statement19. This tool emphasizes transparent and comprehensive reporting, akin to the PRISMA statement26. The PRIOR statement provides a checklist with 27 items covering all steps and considerations involved in planning and conducting umbrella reviews of healthcare interventions, along with an explanation and elaboration document for each item and a flow diagram. However, it is fundamental to distinguish between tools for reporting and those guiding the actual conduct of reviews. The JBI guidance and the Cochrane Handbook are primarily geared towards the conduct of systematic reviews, ensuring methodological rigor, and providing quality assessment tools27,28. On the other hand, the PRIOR statement focuses on the clarity and transparency of reporting. This distinction, between methodological conduct and reporting transparency, is critical for the robustness and comprehension of reviews, a point emphasized in a recent article on best practices for systematic reviews61. While the PRIOR tool offers a novel approach to dealing with heterogeneity in reporting among umbrella reviews, its recent introduction means it has not yet gained widespread adoption in scientific publications. For instance, in our sample of vaccine umbrella reviews, none had yet adopted this checklist. Another guidance on performing and reporting umbrella reviews was also published recently18 and it may also help improve the standards of these evidence syntheses. Concluding comments Achieving universal methods for producing rigorous and reproducible scientific evidence has become a recurrent theme in the scientific community, highlighting a shared desire among researchers for reliable and trustworthy contents, particularly given the exponential growth in the scientific literature. There is a need to summarize the ever-growing evidence from a wide variety of fields. However, this cannot be performed adequately without monitoring how such science is produced, accounting not only for scarcity of methodological rigor, but sometimes even fraudulent content. Detailed discussion of fraud is beyond the scope of this Perspective paper, but there is increasing evidence that fraudulent papers are more frequent than thought64–69 and their inclusion may sometimes affect both systematic reviews and umbrella reviews70. The challenges are particularly emblematic when it comes to public health and epidemiological studies. These studies often include very large and heterogeneous populations and interventions. Appraising the quality of the studies and being able to draw conclusions from them can be a daunting task, even if the data are not fraudulent. The majority of the studies covering public health topics include multiple types of studies and interventions, often on different infectious diseases, so the conclusions made for each domain are limited in their generalizability. Nevertheless, umbrella reviews in this field are still useful because they allow for discussion and comparison to be performed, helping inform selection of future targeted interventions. Vaccination is an example of a topic where contemporary problems arising from misinformation and fake news spreading on social media mean it is of paramount importance to report research with due diligence and attention. This should avoid generating even more confusion and misleading information among healthcare providers who use the scientific literature when preparing guidelines for their practice and the general public, that searches scientific databases for trustworthy evidence. Vaccinology includes a massive number of topics, that spread from the formulation and testing of serums to contain a vast spectrum of infectious diseases, to assessing hesitancy and refusal in the general population, to public health interventions to improve the acceptance rate and accessibility to vaccination services, across very different population groups. It is for large topics such as this that umbrella reviews are a particularly powerful way to synthetize the existing evidence, provide a general picture in the vast ocean of scientific publications in this field and highlight the points of strengths and the gaps in what has already been performed to-date. Supplementary information Peer Review File Description of Additional Supplementary Files Supplementary Data 1 Supplementary Data 2 Supplementary Data 3 Supplementary Data 4 Reporting summary Acknowledgements The work of John Ioannidis is supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford. The work of Rosa Katia Bellomo is funded within the framework of the project “ExACT - European network staff eXchange for integrating precision health in the health Care sysTems” (Grant Agreement n. 823995) funded by the European Commission under the H2020 – Marie Slodowska Curie Action – RISE scheme. Author contributions Both authors contributed to the conception and design of the work, on the acquisition and interpretation of data; RK. Bellomo wrote the original draft and John P. A. Ioannidis substantively revised and edited the submitted version of the manuscript. Both authors approved the submitted version and agreed both to be personally accountable for the author’s own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. Peer review Peer review information Communications Medicine thanks the anonymous reviewers for their contribution to the peer review of this work. A peer review file is available. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary information The online version contains supplementary material available at 10.1038/s43856-024-00679-5. References Associated Data Supplementary Materials Peer Review File Description of Additional Supplementary Files Supplementary Data 1 Supplementary Data 2 Supplementary Data 3 Supplementary Data 4 Reporting summary