Addressing challenges in reviewing e-cigarette evidence: proposing a new ‘Test for Dominance’ in systematic reviews Authors: Garth Reid, Glenna Nightingale, Nason Maani, Mike Smith, Mark Petticrew Published: 2026 ABSTRACT AbstractBackgroundCurrent systematic review evidence on e-cigarettes as a smoking cessation tool is contested. Part of the reason for this is the difficulty of objectively assessing the limited and heterogeneous evidence base.The study aim was to develop a novel approach to testing the validity of systematic reviews and assessing the extent of the dominance of included primary studies.MethodsLeave-one-out meta-analysis was used to measure dominance. The systematic review on e-cigarettes as a smoking cessation tool by Hanewinkel et al. was used as a proof of concept.ResultsThe Hajek et al. study dominates the Hanewinkel et al. systematic review, exerting a disproportionate influence on the overall pooled effect.ConclusionsThis research has identified a gap in current approaches to validity testing in systematic reviews. The Test for Dominance could supplement existing testing, including tests for heterogeneity, sensitivity analysis, and subgroup analysis. When a single study dominates a systematic review, the findings of that review should be interpreted with caution and further randomised controlled trials undertaken to reduce such distortion. Introduction E-cigarettes are an area of intense debate in the public health community, media, and society. In 2023, two large, robust systematic reviews reached very different conclusions about the effectiveness of e-cigarettes as a smoking cessation aid.1,2 One identified that ‘There was high certainty that quit rates were higher in people randomised to nicotine EC [e-cigarettes] than in those randomised to nicotine replacement therapy (NRT)’.1 The other identified that ‘There is limited evidence that freebase nicotine e-cigarettes used with clinical support are efficacious aids for smoking cessation’.2 We investigated why this could have occurred. The first step was to identify all systematic reviews published by June 2024 which investigated the effectiveness of e-cigarettes as a smoking cessation tool. We identified 19 systematic reviews. We examined the methods used in each of the reviews to seek to uncover why some of the reviews of the same, or very similar group of, primary studies had ended up with such different conclusions. As part of our investigation, we scrutinised the meta-analyses in the published systematic reviews. We observed that a single randomised controlled trial3 accounted for much of the pooled effect in all 19 systematic reviews published by mid-June 2024. We termed this finding, the Dominance Effect. We observed that although all 19 systematic reviews had undertaken sensitivity analysis and many had carried out subgroup analysis, none identified the dominance of the Hajek et al. randomised controlled trial.3 We concluded that there was a gap in current approaches to validity testing in systematic reviews and that a novel method needed to be developed that measures the Dominance Effect. The dominant study All systematic reviews on e-cigarettes as a smoking cessation tool, published since 2019, up until 14 June 2024 have included a randomised trial by Peter Hajek and colleagues.3 This is a sizable, meticulously conducted randomised controlled trial. It dwarfs all randomised controlled trials on the topic. Peter Hajek is a contemporary with another academic, Brian May.4 They both undertook their PhDs at the same time. Hajek received his PhD in 1973. May had almost completed his thesis but left academia in 1974 without submitting this.4 As a thought experiment, imagine a parallel universe where Hajek’s career followed that of May. In this universe, it was he who left his PhD manuscript unfinished and instead became the lead guitarist of one of the biggest bands of all time, Queen. Hajek’s 2019 randomised controlled trial would never have been conducted or published and would be absent from the e-cigarette evidence base. We wondered what impact removing Hajek’s large trial would have on the overall evidence base on e-cigarettes as a smoking cessation tool. We are highlighting that the Hajek et al. trial dominates the evidence base, yet reviews of the evidence to date do not report this. To understand how the large effect of a single trial might affect wider systematic review findings, it is important to consider the purpose of systematic reviews and the nature of validity testing. The purpose of systematic reviews is to summarise existing research on a particular topic, reduce uncertainty5 and identify gaps in the literature. Systematic reviews aim to provide insights into a topic area that single studies are unable to achieve. Meta-analyses pool effect sizes from individual studies to uncover the true impact of an intervention, drug, or programme. Validity testing enables researchers to understand the scientific robustness and accuracy of their findings. Validity testing in systematic reviews Handling scientific evidence is challenging. Systematic review methods are well established, and they contain many checks and balances to maintain academic rigour. Cochrane is an international evidence organisation that specialises in systematic reviews. Its methods are recognised as being the gold standard for systematic reviews. The Cochrane Handbook for Systematic Reviews of Interventions6 highlights that undertaking systematic reviews requires a sequence of decisions to be made. They state that ‘Whilst many of these decisions are clearly objective and noncontentious, some will be somewhat arbitrary or unclear. It is desirable to prove that the findings from a systematic review are not dependent on such arbitrary or unclear decisions’. Thus, when carrying out a systematic review and meta-analysis it is important to understand the detail of the methods used by primary studies and consider their impact. A key part of undertaking a systematic review is assessing variability within the included studies and consider whether it is valid to pool their results into a meta-analysis. There are three main types of tests: heterogeneity testing, sensitivity analysis, and subgroup analysis. Heterogeneity assesses and measures the variability of results from a range of different studies. Sensitivity analysis assesses how sensitive the pooled effect is to data limitations and assumptions. Subgroup analysis uncovers subgroups of participants who may differ positively or negatively compared with other groups. A dart board is often used as analogy for validity and reliability testing. Validity testing refers to how close to the bullseye the dart is. Reliability relates to how consistently a dart can be thrown to the same spot. Heterogeneity testing describes the spread of the darts in relation to the bullseye. Sensitivity analyses and subgroup analyses consider the composition of each dart and what impact that might have on the spread of the darts. The Test for Dominance considers the size of each dart and whether there is one particularly large dart, i.e. having an unduly large influence on the pooled effect. The study aim was to develop a novel approach to testing the validity of systematic reviews and assessing the extent of the dominance of included primary studies by deploying ‘Leave-one-out meta-analysis’. Methods Using ‘Leave-one-out meta-analysis’7 to assess the extent of dominance involved performing multiple meta-analyses excluding one study at each analysis. Studies were sequentially omitted, one study at a time, enabling us to assess the impact of the effect of each individual study on the pooled effect. This method helps to identify studies that are outliers and which are dominant in the meta-analysis. For each study, we calculated the effect size of e-cigarettes as a smoking cessation aid. We used a Common Effect model for the leave-one-out meta-analysis to account for heterogeneity between studies. The systematic review by Hanewinkel et al.8 was selected as a case study, because it illustrates the issue of dominance clearly. The findings are presented in Figs 1–3. Meta-analysis from Hanewinkel et al.7 systematic review. In addition, we used the results from the leave-one-out meta-analysis to create a summary figure (Fig. 4) that illustrates the impact of each ‘leave one out’ scenario. In this figure, bars indicate the difference between the effects obtained when all the studies are considered and when one is removed (termed ‘load’). If the difference is negative, i.e. the Common Effect obtained is lower than when all studies are included, the bar is below the x-axis. Dominance in our framework means the study has the largest load and alters the statistical significance of the test. The x-axis is highlighted in red (dotted line)—this is the zero-line where the load associated with each study will be above or below if positive or negative. If the load is zero, then it will be reflected as sitting on the dotted line. In addition to the above, we undertook leave-one-out meta-analysis for all systematic reviews on e-cigarettes as a smoking cessation aid that reported event-level data and included Hajek et al.’s 20193 study. Only 10 of the 19 systematic reviews reported event-level data. We re-analysed all these meta-analyses. Results Leave-one-out meta-analysis of Hanewinkel et al. study From Hanewinkel et al.’s systematic review on e-cigarettes as a smoking cessation aid,8 the original meta-analysis found pooled results comparing their smoking reduction with that of NRT groups to produce an Relative Risk (RR) of 1.57 using the Common Effect model (Fig. 1). With the Bullen et al.9 study removed from the meta-analysis in Hanewinkel et al.’s systematic review, the pooled effect is slightly larger (1.66) than in the original publication (1.57) (Fig. 2). Meta-analysis from Hanewinkel et al. systematic review with Bullen et al.9 study removed. With the Hajek et al.3 paper removed from Hanewinkel et al.’s meta-analysis, the pooled effect becomes 1.17 compared with the original 1.57 reported (Fig. 3). Removing this study3 using leave-one-out meta-analysis resulted in the uncertainty of the effect being increased and the 95% confidence intervals now spanning ‘1’. Meta-analysis from Hanewinkel et al. systematic review with Hajek et al.3 paper removed. When the Bonevski et al.10 study was removed it showed that the pooled effect is slightly larger (1.69) than in the original published meta-analysis (1.57). Removing the Lee et al. study11 resulted in a pooled effect of 1.55. The meta-analyses for those studies are included in the supplementary data that accompanies this article. Overall dominance effect result for Hanewinkel et al. study In summary, illustrating the impact of each ‘leave one out’ scenario highlights the dominance, in the form of a negative load (−0.4), of the Hajek et al. paper (Fig. 4). Summary figure of study dominance. Leave-one-out meta-analysis for all included systematic reviews We found that removing Hajek et al.’s 20193 study reduced the overall pooled effect (using the Common Effect model) in all 10 systematic reviews. In six of the systematic reviews, this reduction was significant and altered the pooled effect so that e-cigarettes were no longer more effective than NRT as a smoking cessation aid. In four, the reduction was not significant, and e-cigarettes remained more effective than NRT as a smoking cessation aid. Importantly, the Hajek et al. study3 was the only study that changed the statistical significance of the pooled effect when it was removed. This illustrates how this study dominates over half of the meta-analyses we re-analysed. Discussion Main finding of this study Our research has shown that the meta-analysis from the Hanewinkel et al.7 systematic review was dominated by a single study.3 Removing this study3 using leave-one-out meta-analysis resulted in the uncertainty of the effect being increased and the 95% confidence intervals now spanning ‘1’. Consequently, we propose that one cannot conclude the weight of evidence indicates that vaping is more effective than NRT for smoking cessation in the Hanewinkel et al. systematic review. This phenomenon was not observed in the study despite extensive sensitivity analysis being undertaken along with a test for heterogeneity. We submit that this illustrates that study dominance is important in systematic reviews and that study dominance has not been considered adequately in systematic review methods. We have proposed that leave-one-out meta-analysis could be used as a method of quantifying study dominance and expanding validity testing in systematic reviews. What is already known on this topic Validity testing is a key part of systematic review methods and is used to determine the robustness of the findings of a systematic review. Different tests examine different elements of validity. Assessing heterogeneity Heterogeneity testing is an inter-study level assessment of variability. It compares the primary studies included in a meta-analysis to uncover how widely they vary in terms of effect size.6 This is important because it explains the precision of the pooled estimate of the meta-analysis. Sensitivity analysis Sensitivity analysis is commonly an intra-study level assessment of validity. Its purpose is to ensure the findings of a systematic review are not dependent on an arbitrary or opaque decision made in the review. Deshmukh12 states that ‘Sensitivity analysis helps in checking the sensitivity of the overall conclusions to various limitations of the data, assumptions, and approach to analysis’. The Cochrane Handbook6 states that: ‘A sensitivity analysis is a repeat of the primary analysis or meta-analysis, substituting alternative decisions or ranges of values for decisions that were arbitrary or unclear’. Subgroup analyses Subgroup analyses are an intra-study level assessment. Subgroup analyses are important in meta-analysis because the effects of an intervention may vary in different population groups. Thus, subgroup analyses help identify subgroups of participants who may experience benefits, or negative outcomes or even adverse effects of an intervention, in comparison with other groups. What this study adds To our knowledge, our research has identified a previously unreported gap in validity testing of systematic reviews and assessing the dominance of included primary studies. To fill this gap, we have described a novel approach. The Test for Dominance complements other tests including the test for heterogeneity and sensitivity analysis and subgroup analysis and could be used alongside those tests in all systematic reviews. We have described a method of assessing dominance in systematic reviews, which operates at the inter-study level, rather than intra-study level. We used leave-one-out meta-analysis. This approach systematically removes each study included in a meta-analysis in a systematic review. It examines what impact removing individual studies has on the pooled effect size. This line of enquiry leads to the generation of the variance of pooled effects and can be used to compare with that from another systematic review. We have shown that some systematic reviews (which did not carry out a Test for Dominance, and instead only carried out tests of heterogeneity and sensitivity analyses and subgroup analyses) missed a key methodological weakness in the review they were undertaking. Consequently, the reliance on a single study has not been observed or considered in reaching a conclusion about the findings. The Test for Dominance could supplement existing validity tests. It complements other approaches and uncovers a different type of inter-study level variability. This will give systematic reviewers a fuller understanding of potential methodological weaknesses in their study and enable them to reach more robust conclusions when making a judgement on their scientific findings. In terms of e-cigarette evidence, our analysis has identified that the results from some systematic reviews (published by mid-June 2024) that have explored the role of e-cigarettes as a smoking cessation tool may be less certain than previously thought and should be interpreted with greater caution. Further randomised controlled trials examining the effectiveness of e-cigarettes as a smoking cessation tool are needed to provide a more balanced picture of the issue. There needs to be greater diversity of randomised controlled trials so that when their findings are pooled the evidence base is not so reliant on a single study. While a single randomised controlled trial can be robust and is valuable, more than one study is needed to provide a diverse and robust pooled effect. We propose that our dominance test illustrates that the Hanewinkel et al. systematic review and meta-analysis was reliant on a single dominant study. This is one example of many across systematic reviews and meta-analyses in which the undue influence of a study may have shaped the overall result. It is important to add that our research does not call into question the validity of individual randomised controlled trials that have explored the effectiveness of e-cigarettes as a cessation tool. So, Hajek et al.’s3 conclusion that ‘E-cigarettes were more effective for smoking cessation than nicotine-replacement therapy, when both products were accompanied by behavioural support’, is scientifically appropriate, based on their randomised controlled trial. However, more caution and more information may be needed in relation to some statements on the strength of the evidence base in this area. Statements on the strength of the evidence base of e-cigarettes as a smoking cessation tool could be usefully reviewed and, in some cases, reevaluated. Our research has uncovered that, in reality, for some systematic reviews the evidence on e-cigarettes as a smoking cessation tool could be unduly skewed by a single study.3 Further primary research would enable meta-analyses on this topic to be more robust. Limitations of this study This is the first description of the Test for Dominance, so further work needs to be done to test the hypothesis. This could include a simulation study as well as an empirical investigation to verify the method. A second limitation is that like other validity tests, the Test for Dominance examines the uncertainty of the pooled effect in a meta-analysis. It does not assess the evidence of an effect versus evidence of no effect. A third limitation is that the Test for Dominance does not assess the strength of individual randomised controlled trials within a systematic review and meta-analysis. Existing methods would need to be used to undertake that work. A fourth limitation is that the Test for Dominance being similar to the Test for Heterogeneity could lead to a perception of duplication. However, they examine different elements of variability, with heterogeneity assessing the spread of effects of individual studies in relation to the pooled estimate, whereas dominance explores the impact an individual study has on the pooled estimate. A final limitation of the Test for Dominance is that it does not address the issues pertaining to the transferability of findings from randomised controlled trials to real-world settings. Supplementary Material