Effect size and event rate estimation in neurocritical care randomised clinical trials: a protocol for a systematic review Authors: Gentle Sunder Shrestha, Nuanprae Kitisin, Phil Talbot, Natalie Linke, Jonathon D Taylor, Camila R Battistuzzo, Ary Serpa Neto, Alisa Higgins, Toby Jeffcote, D James Cooper, Andrew Alexander Udy Published: 2025 ABSTRACT AbstractIntroductionEffect size and event rate estimation is necessary for sample size calculation in randomised clinical trials. Overestimation of the effect size and event rate can lead to inadequately powered studies and increased probability of false negative results. This is common in trials involving critically ill patients. However, such overestimation has not been systematically evaluated in trials involving neurocritical care. We aimed to conduct a systematic review of published randomised clinical trials involving critically ill neurological patients, to determine the accuracy of effect size and event rate estimation.Methods and analysisWe will review randomised clinical trials involving adult critically ill neurological patients that were published from 2015 onwards in selected clinically useful and high-impact journals. We will include randomised clinical trials reporting a binary or time to event outcome, using two study groups, and a superiority design testing the efficacy of diagnostic, monitoring, therapeutic or process interventions. All eligible studies must report an estimated event rate in the control group and estimated effect size. All relevant studies will be identified through database searches. All study selection and data extraction will be conducted by two independent reviewers. We will use a random-effects model for pooling data. This review will be conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guidelines. Accuracy of effect size and event rate estimation will be evaluated by comparing the estimated and observed values. The association between the accuracy of the individual randomised clinical trial effect size and event rate estimation and rejection of the null hypothesis will be evaluated using logistic regression analysis. Multivariable linear regression analysis will be used to explore the factors associated with accuracy of effect size and event rate estimation. In addition, we will perform subgroup analysis by impact factor of the published journals, sample size of the studies and risk of bias.Ethics and disseminationAs this systematic review will use data from previously published studies, it does not require ethics approval. Findings of this systematic review will be published in a peer-reviewed journal and will be presented at specialty-based conferences. The study will be included in the higher degree research thesis of the primary author.PROSPERO registration numberCRD420251106394.Keywords: Adult intensive & critical care, Research Design, Stroke, Brain Injuries STRENGTHS AND LIMITATIONS OF THIS STUDY This systematic review will evaluate all major randomised clinical trials from 2015 onwards involving critically ill neurological patients. This systematic review will evaluate all major randomised clinical trials from 2015 onwards involving critically ill neurological patients. We will review randomised clinical trials with a superiority design and will exclude equivalence and non-inferiority trials. We will not review phase 2 studies, non-randomised trials, observational studies, case series and case reports. We will review randomised clinical trials with a superiority design and will exclude equivalence and non-inferiority trials. We will not review phase 2 studies, non-randomised trials, observational studies, case series and case reports. Studies with a documented estimated effect size and control event rate will be included. Studies with a documented estimated effect size and control event rate will be included. We will exclude paediatric studies and those not in English. We will exclude paediatric studies and those not in English. Introduction Randomised clinical trials are often considered the gold standard for assessing the effectiveness of a treatment or intervention.1 However, the majority of critical care randomised clinical trials do not demonstrate positive effects.2 3 Several factors may be responsible, including heterogeneity of treatment effect, ineffective interventions and inadequately powered studies.2 Calculation of the required sample size for a clinical trial depends on significance level, power, estimated baseline event rate and the estimated effect size of the intervention. Estimated effect size is the magnitude of difference in the outcome of interest between the groups that the randomised clinical trial is designed to detect reliably.4 Effect size estimation is often based on existing data, biological plausibility or the minimal difference that is considered clinically important. As the other parameters for sample size calculation are typically fixed, this often renders effect size estimation vulnerable to inflation. Overestimation of the effect size can lead to trials with inadequate sample sizes and thus false negative results, due to a type II error.5 This can lead to the withholding of interventions that are actually helpful. Also, underestimation of effect size can lead to a very large sample size with consequent ethical concerns and overuse of resources. Effect size overestimation is common in trials involving critically ill patients and also in other disciplines.5,8 The recent Consolidated Standards of Reporting Trials (CONSORT)-Outcomes 2022 Extension has elaborated the sample size section of the CONSORT 2010 statement and has recommended defining and justifying the effect size and target difference while reporting clinical trials.9 Trials involving critically ill neurological patients may be particularly vulnerable to effect size overestimation, due to a lack of baseline data in these cohorts and historical difficulties in completing patient enrolment.10 As such, our primary aim is to systematically assess the accuracy of effect size and event rate estimation in randomised clinical trials involving such patients, with a view to informing future trials activity. Objectives To assess the accuracy of effect size and event rate estimation in randomised clinical trials involving adult critically ill neurological patients and its association with rejection of the null hypothesis. We will also evaluate factors associated with accuracy of effect size and event rate estimation. Methods and analysis Study design We will perform a systematic review of the existing literature. This protocol is reported according to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) statement (online supplemental 1).11 The protocol is registered in the PROSPERO database. The study results will be reported according to the PRISMA statement.12 Eligibility criteria Types of studies We will include two-arm parallel group randomised clinical trials with superiority design and with a documented estimated effect size and control event rate in adult critically ill neurological patients. We will report the proportion of studies that have provided estimated effect size and control event rate. Trials with equivalence and non-inferiority designs and those with a sample size <50 will not be included. We will exclude studies with cluster randomisation, phase 2 studies, non-randomised trials, observational studies, case series and case reports. Studies that are mentioned to be of phase 2 design in either title, abstract or methodology will be considered as phase 2 studies. Animal studies and those not published in English will be excluded. Studies conducted outside intensive care unit (emergency department, postanaesthesia care unit, stroke unit and rehabilitation facilities) will also be excluded. Types of participants We will include studies involving adult participants aged 16 years and above. Studies involving ICU patients with stroke (acute ischaemic stroke, intracerebral haemorrhage and subarachnoid haemorrhage), traumatic brain injury, status epilepticus (including refractory and super refractory status), meningitis, encephalitis and postcardiac arrest (studies enrolling patients that remain unconscious after cardiac arrest and with neurological endpoint as primary outcome) will be included for analysis. Types of interventions We will include studies exploring the effectiveness of diagnostic, monitoring, therapeutic or process interventions. Types of outcomes measured Studies reporting a dichotomised patient-centred primary neurological outcome, such as mortality, functional status, event-free survival or quality of life will be included. We will also include studies reporting time to event outcome. However, we will not include studies reporting continuous outcome or those reporting non-neurological end-points such as duration of mechanical ventilation or incidence of ventilator-associated pneumonia. Studies examining intermediate and physiologic outcomes will also be excluded. Data sources Studies will be identified through searches of the following databases, from January 2015 until current: Ovid MEDLINE, Ovid Embase and Cochrane Central Register of Controlled Trials. Search will be limited to the clinically useful journals (CUJ) and 16 high impact journals. The CUJ, previously known as core clinical journals, includes a list of journals that were selected using evidence-based methodology and is described elsewhere.13 The CUJ list has been used by systematic reviews.14 15 The list of 16 high impact factor journals is based on those used by previous systematic reviews involving critically ill patients.2 16 We will screen previous systematic reviews for any relevant citations. We will also search trial and study registers for published study protocols to obtain any additional necessary information. Details of the search strategy are presented in (online supplemental 2). Data collection and analysis Study selection process Studies identified using the search strategy will be uploaded into Covidence and duplicates will be removed. Abstracts will be independently reviewed by two authors and will be categorised as ‘exclude’, ‘include’ or ‘maybe’. Abstracts within the latter two categories will be advanced to screening. Full texts of the studies requiring screening will then be uploaded to Covidence and reviewed by the same authors for final inclusion. Disagreement in study selection at any point will be resolved via consensus or referral to a third independent author. Additional information will be obtained from the accompanying or published study protocol, as required. A final list of studies will then be collated for further analysis. Data collection process Data will be extracted independently by two authors using a data extraction template. The following general characteristics will be extracted: type of research organisation (academic or contract), funding details (industry sponsored, investigator initiated with industry co-funded, investigator initiated with non-profit funded or not funded), intervention tested (diagnostic, monitoring, therapeutic, process or other modalities) and the impact factor of the journal in which the study was published (as per the 2023 Clarivate Journal Citation Reports).17 The following data related to study design will be collected: blinding of interventions (blinded or open label), presence of event adjudication committee, estimated and observed effect size, estimated and observed event rate (in the control group), rationale for estimation of effect size and event rate (based on minimal clinically important difference, pilot study, phase 2 study, meta-analysis, multiple randomised clinical trials, single randomised clinical trial, multiple observational studies, single observational study, survey, expert opinion, unpublished data or none), estimated sample size, estimated power, diagnostic category of participants (acute ischaemic stroke, intracerebral haemorrhage, subarachnoid haemorrhage, traumatic brain injury, status epilepticus, meningitis, encephalitis or postcardiac arrest) and whether the study had a composite outcome as the primary end point. The following parameters related to the conduct of the study and study outcome will be obtained: number of study sites, number of countries involved, number of participants randomised, whether the trial was terminated early, whether the sample size was modified during the study and whether the study rejected the null hypothesis. Extracted data will be compared and any disparities will be resolved via discussion between the two reviewers, or if needed, the third reviewer will be involved. Outcome data Reported hazard ratio (HR), odds ratio (OR), relative risk, risk ratio or difference in the primary outcome will be considered as the effect size of the study. If the difference in incidence of outcome is expressed as absolute difference, it will be converted to a relative difference from the baseline incidence. For trials reporting event-free survival as the primary outcome, event rate will be calculated by subtracting event-free survival from 1. For studies reporting time-to-event as the primary outcome, estimated and observed event rates will be extracted for the same time period, with the time defined for estimated event rate as the reference. If the time period varied between estimated and observed event rate, for the purpose of harmonisation, observed event rate will be calculated from the Kaplan-Meier curve corresponding to the time period of the estimated event rate using the digitizer software WebPlotDigitizer, V.5.2.18 If the measure of expected and observed effect size varied, for the purpose of harmonisation, the expected effect size will be converted into the measure of the observed effect size using statistical formulae (online supplemental 2).6 The expected and observed event rates in the control group will be extracted. The null hypothesis will be considered to be rejected if the study reports statistically significant outcomes as per the predefined p value. Risk of bias assessment The Cochrane risk-of-bias tool for randomised trials (RoB 2) will be used to assess the risk of bias in the randomised trials.19 Two authors will independently assess the parameters that can introduce bias in the studies: process of randomisation, deviation from intended intervention, missing data, measurement of outcome and selection of the reported result.20 Any disagreement will be resolved by discussion or inclusion of a third author. The studies will be categorised as ‘low’, ‘some concerns’ or ‘high’ risk of bias. Data synthesis Descriptive analysis Categorical variables will be presented as numbers with frequencies and continuous variables will be presented as median with IQR or mean±SD. Statistical analysis Accuracy of effect size estimation will be calculated as: (observed effect size – estimated effect size)/estimated effect size. Similarly, the accuracy of event rate in the control group will be calculated as: (observed event rate – estimated event rate)/estimated event rate.6 Absolute values will also be calculated for estimating accuracy of effect size and event rate, which will be calculated as (observed effect size – estimated effect size) and (observed event rate – estimated event rate), respectively. The association between accuracy of effect size and event rate and rejection of the null hypothesis will be evaluated using logistic regression analysis. Multivariable linear regression analysis will be used to explore the association between accuracy of effect size and event rate, and the characteristics of trials: type of research organisation, type of funding, type of intervention tested, impact factor of the publishing journal, blinding of interventions, presence of event adjudication committee, provided rationale for estimation of effect size and event rate, estimated sample size, estimated power of the study, diagnostic categories of the participants, composite outcome as the primary end point, number of study sites, number of countries involved, number of participants randomised, whether the trial was terminated early and whether the sample size was modified during the study. Bland-Altman plots will be created to visually inspect agreement between estimated and observed values and for estimation of proportional bias. Sensitivity analysis We will conduct sensitivity analysis for the studies published in top-tier journals (The New England Journal of Medicine, The Lancet and Journal of the American Medical Association), for the studies with low risk of bias, studies with larger sample size (above the median sample size of the studies included in the systematic review) and the studies that have implemented minimal clinically important difference for estimation of effect size. Confidence in cumulative evidence Not applicable. Ethics and dissemination As this is a systematic review of published studies, ethics approval is not required. We will publish the results in a peer-reviewed scientific journal. Results from this systematic review may be presented in local academic meetings and conferences prior to full publication. The systematic review will also be included as a chapter in the primary author’s higher degree research thesis. The dataset will be made available to the researchers based on reasonable request. Patient and public involvement We have involved two patients, PT and NL, from the stage of protocol development. They have made comments and suggestions on the draft of the protocol and have joined as coauthors. We will continue their involvement as per the ACTIVE framework and will report as per the GRIPP2 (guidance for reporting involvement of patients and public in research, version 2).21 22 Limitations This systematic review will exclude equivalence and non-inferiority trials. We will not include phase 2 studies, non-randomised trials, observational studies, case series and case reports. We will not include paediatric studies and those not in English. As we have limited our search to CUJ and the list of 16 high impact factor journals, we might not have included some eligible neurocritical care studies. Supplementary material Acknowledgements The authors would like to thank the librarian, Lorena Romero, at the Ian Potter library of the Alfred Hospital, Melbourne, for her assistance in developing the search strategy. Footnotes Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors. prepub: Prepublication history and additional supplemental material for this paper are available online. To view these files, please visit the journal online (https://doi.org/10.1136/bmjopen-2025-109289). Provenance and peer review: Not commissioned; externally peer reviewed. Patient consent for publication: Not applicable. Patient and public involvement: Patients and/or the public were involved in the design, conduct, reporting or dissemination plans of this research. Refer to the Methods section for further details. References Supplementary Materials