Umbrella-Review, Evaluation, Analysis and Communication Hub (U-REACH): a novel living umbrella review knowledge translation approach Authors: Corentin J Gosling, Samuele Cortese, Joaquim Radua, David Moher, Richard Delorme, Marco Solmi Published: 2024 ABSTRACT AbstractSystematic reviews and meta-analyses have become crucial for evidence-based decision-making in recent decades. However, it is common for the results of multiple reviews on the same topic to be inconsistent, and it is widely recognised that the results of the reviews are not always effectively communicated to healthcare professionals and the lay public. This manuscript proposes a strategy to summarise and communicate the findings of previous systematic reviews and meta-analyses to wider audiences. The proposed approach couples the findings of umbrella reviews with the creation of open-access online platforms that present the results of these umbrella reviews in an accessible way to various stakeholders. The key potential methodological avenues of this approach are presented, and specific examples from the author’s own works and those from other teams are provided. An accompanying website (https://u-reach.org/) has been designed to present this Umbrella-Review, Evaluation, Analysis, and Communication Hub (U-REACH) approach and to overcome the technical challenges associated with this type of project (by sharing the code used to build existing U-REACH projects). The present document is intended to serve as a methodological and technical guide for the creation of large-scale projects designed to synthesise and disseminate scientific information to a broad audience. Systematic reviews and meta-analyses (SR/MAs) are essential tools for summarising the nearly 2 million annual scientific publications.1 However, their real-world impact remains limited for several reasons. First, SR/MAs usually focus on a narrow set of interventions and/or outcomes, which do not necessarily match with clinical or research needs. Second, overlapping SR/MAs, that is, focusing on the same combination of Participants, Intervention/exposure, Comparators, Outcomes (PICO), can lead to different conclusions, providing readers with conflicting information.2 Third, the results of the SR/MAs are not always effectively communicated to researchers, healthcare professionals or the lay public. These reasons partly explain why SR/MAs are inconsistently used in the real-world, for instance, in clinical guidelines.3 Umbrella reviews (URs) can address some of these limitations by summarising data from multiple SR/MAs on a broad topic, and by providing a careful assessment of the strength and/or quality of the evidence to guide readers to the current best evidence. However, making UR findings accessible to various stakeholders remains a critical need. We propose a novel approach, referred to as Umbrella-Review, Evaluation, Analysis and Communication Hub (U-REACH), aimed at quantitatively summarising, appraising and effectively disseminating evidence pooled in URs, adopting a living (ie, continuously updated) evidence synthesis approach. We suggest that coupling the findings of a living UR with the creation of open-access online platforms—presenting the results of the UR in an accessible way—is crucial for knowledge dissemination. Here, we describe the proposed steps to build a U-REACH project (figure 1). Overview of the U-REACH approach. Concrete examples of ongoing U-REACH project are available online: https://u-reach.org. U-REACH method Umbrella review: identification of SR/MAs Each U-REACH project should perform a comprehensive search of available SR/MAs on a given topic. A key a priori decision is whether to limit the search to SRs with MA (allowing quantitative evaluation) or to also include SRs without MA (allowing broader mapping but resulting in the inclusion of qualitative conclusions). When multiple SR/MAs are available for a given PICO, it is recommended that only one be presented in the primary analysis.4 This selection should be done with consideration of factors such as (in no particular order): the SR/MA with inclusion criteria that best match the needs of the clinicians/people with lived experience; the SR/MA that promotes equality, diversity and inclusion; the SR/MA with the largest number of studies, the most recent SR/MA; the SR/MA with the highest methodological quality; the SR with MA over the SR without MA; or a network MA over pairwise MA. Different U-REACH projects may adopt varying selection processes, each with its own set of advantages and disadvantages. For example, choosing the highest-quality SR/MA might exclude a more recent, larger one with slightly lower quality. Regardless of the selection strategy employed, it must be made a priori, transparently reported and justified. If authors of the U-REACH project wish to assess the consistency of SR/MAs on the same PICO in a secondary analysis, they can rely on existing software.5 Evaluation of quality of included evidence Any U-REACH project should critically assess the methodological quality of retained SR/MAs and their primary studies. For SR/MAs, authors of U-REACH projects should use established tools, such as A MeaSurement Tool to Assess systematic Reviews-2 (AMSTAR-2, providing a score reflecting the methodological quality of SR/MAs),6 or the AMSTAR-PLUS (with a score reflecting the methodological quality of SR/MAs plus that of primary studies).7 For the primary studies included in SR/MAs, we advise obtaining this information directly from the SR/MA report when the assessment was adequately performed (ideally at the outcome level, using a double-blind scoring on standard tools, such as the Cochrane Risk of Bias tool V.2).8 It is not uncommon for the tools used to assess the risk of bias in individual studies to vary across SR/MAs. In our view, the optimal approach to address this issue is to meticulously delineate the specific biases of interest in the protocol (ie, those that will be used in data analysis). Consequently, despite the potential for variability in the tools employed, a systematic extraction of a common set of biases can be achieved across tools. Where certain SR/MAs have failed to assess some risks of bias as the tool used did not encompass these, we advise that authors of U-REACH should perform such an assessment. Analysis and/or assessment of SR/MS results When synthesising results of previous SR without MA, a narrative summary can be produced (eg, relying on the synthesis without meta-analysis guidelines).9 When synthesising results of previous MAs, one could extract data from primary studies included in each SR/MA and re-perform the meta-analytical calculations.10 If the included MAs used similar statistical approaches (eg, all used a random-effects model), another option is to directly extract the reported meta-analytical results. This approach, besides being more feasible, allows the inclusion of pooled estimates from network meta-analyses (NMAs) even when the raw data are not publicly available. There are statistical software packages for both methods.11 Consideration around the quality of available SR/MAs and feasibility may inform the analytical approach that should be adopted. Alongside statistical pooling, U-REACH projects should assess the certainty of evidence from SR/MAs. For interventional evidence, the standard approach is the Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework.12 Due to the inherent subjectivity of GRADE, authors of large umbrella reviews have raised concerns about grading large amounts of evidence. To address this, some authors have proposed an objective scoring of GRADE criteria, which aligns with tools such as Confidence in Network Meta-Analysis (CINeMA), commonly used in SR/NMA.13 However, the objective approach should not be seen as equivalent to standard GRADE. For observational evidence, Ioannidis’ algorithmic approach has been widely adopted.14 The criteria for assessing certainty should be tailored to meet the needs of each UR. Communication hub to present the results of URs Key principles in terms of implementation in U-REACH platforms are: Living evidence synthesis: Similarly to the approach taken for living systematic reviews,15 a ‘living evidence synthesis’ methodology is required for U-REACH projects, to ensure the platforms disseminate up-to-date information even when new evidence alters the conclusions previously reached. The frequency of updates must be determined on a case-by-case basis in accordance with the rate of publication of the synthesised field. A U-REACH project may transition out of the living mode, either temporarily or permanently, in the event that (1) the research question is no longer a priority for decision-making, as determined by the relevant stakeholders, or (2) a reasonable level of certainty has been reached, or (3) research that might impact the conclusions of the review, is no longer emerging. Open science: U-REACH online platforms should follow open science principles regarding citation standards, registration, transparency of data, analytical code and research material (platform) to allow replication. We are currently developing a new registration checklist adapted to U-REACH projects using a Delphi process (a preliminary version is available online: https://u-reach.org/). Co-designing the platform: U-REACH platforms should be co-designed with a broad range of stakeholders, including method experts, clinicians, policymakers and people with lived experience, depending on the specific topic of the U-REACH project. Delphi processes can be used to reach a final consensus on the platform’s features and content, and validated quality tools (eg, DISCERN16) can objectively measure the quality of the information contained in the platform. Stakeholder-specific U-REACH interfaces: U-REACH interfaces should be usable by researchers, decision-makers, clinicians, or people with lived experience, and adapted accordingly, as follows: For researchers: U-REACH platforms should provide more ‘technical’ information about study variables/results, such as the risk of bias of individual studies, total sample size, heterogeneity/inconsistency indicators, indicators of potential reporting bias and the magnitude of the pooled effect size. For policymakers and clinical practice guideline developers: U-REACH platforms should have a dedicated interface allowing users to easily filter the results of the URs by PICO and other desirable clinical or methodological components. Graphical representations of the results (eg, forest plots) can support knowledge translation. The link to individual studies included in SR/MAs will allow critical assessment of the evidence source. For clinician–patient shared clinical decision-making: U-REACH platforms of interventions in healthcare may have a section where the efficacy and safety of different interventions that are recommended by clinical guidelines are visualised reflecting clinicians’ and patients’ preferences, and history of response to specific treatments, and certainty of evidence. For the lay public including persons with lived experience and family members/carers: U-REACH platforms are opportunities to include psycho-educational strategies. Producing a narrative summary of the results of the UR is also an important step to ensure dissemination to a wide audience without a scientific background. Tools can be used to ensure the results’ summary is adapted to the lay public (eg, https://www.thewriter.com/tools/readability). Conclusion The U-REACH approach, by combining for the first time a broad, rigorous, systematic living evidence synthesis framework, open science and interactive platforms for diverse stakeholders, aims to improve access to and uptake of evidence for key stakeholders. To facilitate the understanding and dissemination of this approach, readers can consult our website (https://u-reach.org/), presenting (1) examples and programming code of some U-REACH projects, (2) a preliminary protocol checklist and (3) an Open Science Framework folder centralising U-REACH protocols.