Experiences from conducting systematic reviews of systematic reviews Authors: Wilhelmus Johannes Andreas Grooten, Tony Bohman, Carina Bostrom, Åsa Dedering, Marie Halvorsen, Roman P Kuster, Lena Nilsson‑Wikmar, Christina B Olsson, Graciela Rovner, Elena Tseli, Eva Rasmussen‑Barr Published: 2026 ABSTRACT AbstractThis commentary discusses recent experiences from conducting two systematic reviews (SRs) of SRs. We address the strengths and limitations of conducting an SR of SRs in the field of evidence-based medicine in musculoskeletal health. Despite methodological challenges, we believe that SRs of SRs add value to the research community.Keywords: Physical therapy, Exercise therapy, Rehabilitation, Chronic musculoskeletal pain, Umbrella review, Evidence-based medicine Introduction Eleven researchers within our research group at the Karolinska Institutet [1] in the musculoskeletal research area have recently conducted two systematic reviews (SRs) of SRs of randomized controlled trials (RCT) regarding the effect of various exercise types on pain and disability levels in patients with chronic low back and neck pain [2, 3]. The rationale for applying this methodology was to summarize and establish the certainty of evidence on the efficacy of various exercise therapies commonly proposed for these conditions since different SRs show different results. With our experiences fresh in mind of conducting these two SRs of SRs, we find it important to discuss the strengths and limitations of conducting an SR of SRs. The value of SRs of SRs should be seen from the perspective of recent discussions [4, 5] that SRs are dependent on the data quality of the original studies included, which could significantly limit their validity. Considering this, determining the evidence through an SR of SRs might thus be even more challenging and raise concern about their added value in evidence-based medicine. Strengths with SRs of SRs There are multiple reasons why SRs of SRs have a value. In 2019, the number of SRs was estimated to 80 SRs published daily [6] and is constantly rising with nearly 113 SR in 2022 [7], which makes it difficult for researchers, clinicians, and policymakers to keep updated. Thus, SRs of SRs enable easy access to a summary of the available literature. Moreover, by merging the results from several SRs, preferably by using meta-analyses, the certainty of evidence in a specific area becomes clearer [8]. For example, the results of one SR may show no significant difference between two interventions, while merging the results of several SRs might show a more granular picture. In addition, the findings from SRs of SRs may serve as an entrance point for clinicians and decision-makers in healthcare fields when developing new clinical guidelines. SRs of SRs can also show what has already been studied and thus may form a base for the recommendation that there is no need to further study certain interventions or, in contrast, that some interventions need additional investigations. To continue to study interventions that already have shown a high certainty of evidence might mean a waste of time, money, and, importantly, the unnecessary use of research participants, which is also an ethical dilemma for researchers. On the other hand, even if some interventions may be considered well studied, it still may be of interest to further investigate these with advanced methodologies, such as meta-syntheses or network analyses to determine the effect size of the interventions. Limitations with SRs of SRs As with every methodology, SRs of SRs entail some limiting factors. Firstly, SRs of SRs can be seen as a sort of whispering game, and participant data can be altered during the processing of three publications (original study, SR, and SR of SRs), which might lead to so-called error propagation. The accuracy of the SR of SR findings is of course dependent on the accuracy and reporting quality of the original study data. When conducting an SR of SRs, it should not be necessary to go back to the original RCTs, but for the purpose of our SRs of SRs, we felt compelled to do so as some of the included SRs showed baffling results. It is important to be sure that the data are correct, and we encourage authors of SRs to increase their reporting accuracy using existing guidelines such as TIDier and CERT [9, 10]. Moreover, journal editors also have a responsibility that the data reported in their journals are correct and that the SRs follow the Prisma 2020 guidelines. As reviewers, we often have very short time frames to complete our review of a submitted manuscript. For SRs and especially SRs of SR, this means only very limited time to check the data and we suggest that the review time should be adjusted to the type of article submitted. Currently, artificial intelligence tools for data extraction are developing and perhaps reviewers could use them in the future to check the accuracy of the data in the tables. However, the available tools are not yet reliable and still require extensive human validation [11]. The value of the SR of SRs is evidently directly related to the quality of the included SRs. In our SR of SRs for low back pain, 36% of the included SRs were of very low-to-critically low quality, and in our SR of SRs for chronic neck pain, this was true for 52%. This, of course, relates to what appraisal tool is used [12], and the fact that the assessment of risk of bias is subjective. We discussed whether authors of SRs of SRs should exclude SRs with low quality to increase their own study quality. It is suggested that bringing forward SRs of SRs on only high-quality SRs is a passable way to provide valid evidence to clinicians [8, 13]. On the other hand, excluding SRs with lower quality might increase the risk that certain topics will be omitted because no high-quality studies and SRs have yet been published. This is likely to be the case for new, emergent topics. The inter-rater reliability in study selection, data extraction and quality assessment when performing an SR of SRs can be a challenge. In our SRs of SRs, we worked in pairs of two and discussed disagreements in a consensus dialogue with a third researcher and/or with the entire research group. For our SRs of SRs, it was necessary to work with five pairs due to the amount of data, and this could have introduced some bias. Working in pairs is a strength and is recommended when conducting SRs [14], but there might still be some inter-assessment variabilities between the pairs. Therefore, consensus discussions in the full group were necessary, although very time-consuming. Moreover, we used the critical appraisal tool AMSTAR-2 [15] for the included SRs as recommended by Cochrane. One limitation with AMSTAR-2 is that comparison of the study quality between different SRs could be problematic since different reviewer pairs could appraise methodological flaws and weaknesses differently. Hence, there’s a need to conduct piloting assessments and we recommend having multiple consensus discussions in the full research group. In addition, neither AMSTAR-2, nor GRADE allow for a downgrade of the level of evidence due to obvious errors in the included SRs. This became evident in the process of extracting data from the included SRs when we, by chance, discovered several errors in the reviews compared to the original studies. In our SRs of SRs, we detected large overlaps for some exercise types, e.g., motor control exercises and Pilates in low back pain, and Traditional Chinese Exercise and Yoga in neck pain. An overlap means that at least two published SRs include the same original article or multiple articles. Authors should be very careful in making conclusions on effectiveness, evidence or health policy when there are more SRs than original studies published on a specific topic [3]. We encourage future authors of SR of SRs to handle this issue appropriately using the methods proposed by Pieper et al., 2014, in which a “corrected covered area (CCA)” is calculated based on the frequency of repeated occurrences of the same paper in the SRs included in the SR of SRs [16]. Still, authors should be aware that one large RCT included in several SRs could influence the point estimates of the included MAs, affecting the overall results of the SR of SRs to a large extent. A major challenge for SRs of SRs is to summarize populations, control groups, interventions, timelines, or outcomes described in different ways in SRs. This becomes particularly evident if the same original study is included in several SRs with different descriptions of the population and intervention. Therefore, following the PICOT structure and the current reporting guidelines, such as proposed by the EQUATOR Network is important to get the included data to be as homogenously described as possible [17]. If a systematic review is not conducted in compliance with these or the PRISMA 2020 guidelines, it is difficult to reach confidence in the results. We cannot interpret the findings correctly and trust significant findings of a systematic review if the quality of the review is substandard. This will affect clinical practice and healthcare policy to a large extent. We thoroughly discussed the feasibility of summarizing the data using a meta-analysis and arrived at different conclusions. In our SR of SRs focusing on exercises for low back pain, we conducted solely a narrative synthesis due to the large heterogeneity among included SRs. However, in the SR of SRs addressing exercises for neck pain, we conducted both a narrative synthesis and a meta-analysis. The narrative analysis encompassed all SRs, whereas the meta-analysis was limited to SRs with sufficiently homogeneous data. We found that the findings from our meta-analysis strengthened the findings from the narrative analysis. We strongly believe that a narrative analysis has great value, but if feasible we advise future authors of SRs of SRs to consider performing a meta-analysis that could be used to strengthen the results and to follow the guidelines for analyzing heterogeneity [14, 18], even if it consists of a sub-sample of the included SRs. To establish the certainty of evidence on the efficacy of exercise interventions, our SRs of SRs were limited to SRs including predominantly RCTs. However, if one wants to investigate additional aspects of exercise interventions such as mechanisms of action explaining intervention effects or what psychological and psychosocial factors are of importance for a successful intervention outcome, other study types could provide equally valuable information. Reviews on these related topics should thus not be limited to RCTs. Conclusion SRs of SRs can be an important tool to analyze the certainty of evidence in a particular research field. They provide a detailed overview of the available literature, might identify research gaps, and could serve as an entrance point when developing new clinical guidelines. However, the conduction of SRs of SRs is challenging. It is therefore important that authors consider data verification to limit error propagation (from the original study to SR to SR of SRs), to define how to deal with low-to-critically low-quality SRs, and to assess overlap (multiple SRs including the same original studies). We also recommend working in pairs but setting off or planning enough time for consensus discussions in the group of authors. Future review articles on this topic might also consider including a broader spectrum of study types to summarize evidence on potential mechanisms or psychological and psychosocial factors of importance for successful interventions. Acknowledgments Not applicable. Abbreviations Assessing the Methodological Quality of Systematic Reviews Corrected Covered Area Enhancing the QUAlity and Transparency Of health Research Systematic Review Population, Intervention, Control group, Outcome and Time Randomized Controlled Trials Authors’ contributions The comment was initiated by WJAG, ER-B and LN-W and wrote the first draft. CB and RPK took the lead for a second version of the comment, including TB and ÅD. After that, a group discussion was held and MH, CBO, GR, and ET finished the comment, and all authors approved the final version. Funding Open access funding provided by Karolinska Institute. No funding was received. Data availability No datasets were generated or analysed during the current study. Declarations Ethics approval and consent to participate Not applicable. Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement No datasets were generated or analysed during the current study.