Optimal weighted tests for replication studies and the two-trials rule

David S. Robertson, Thomas Jaki · 2025

Read Optimal weighted tests for replication studies and the two-trials rule by David S. Robertson, Thomas Jaki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21708.txt.

Introduction

David S. Robertson and Thomas Jaki tune the statistics of replication studies, the follow-up trials science uses to check whether a result was real. Drug approval formalizes replication as a two-trials rule. When several hypotheses compete at once, the familywise error rate must stay controlled, usually by the conservative Bonferroni split of the significance budget.

Weighting that split by each hypothesis's prior importance can buy power. The authors choose the weights for a replication trial optimally, using the original study's results to aim at maximizing the chance of rejecting at least one false hypothesis. The gains over the unweighted procedure can be substantial, and the method stays robust when effect sizes drift between the original and the replication, a common worry. The result is a principled way to let a first study teach its successor where to look.

Catalog record

CREATORDavid S. Robertson, Thomas Jaki
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimal weighted tests for replication studies and the two-trials rule David S. Robertson, Thomas Jaki Abstract Replication studies for scientific research are an important part of ensuring the reliability and integrity of experimental findings. In the context of clinical trials, the concept of replication has been formalised by the ‘two-trials’ rule, where two pivotal studies are required to show positive results before a drug can be approved. In experiments testing multiple hypotheses simultaneously, control of the overall familywise error rate (FWER) is additionally required in many contexts. The well-known Bonferroni procedure controls the FWER, and a natural extension is to introduce weights into this procedure to reflect the a-priori importance of hypotheses or to maximise some measure of the overall power of the experiment. In this paper, we consider analysing a replication study using an optimal weighted Bonferroni procedure, with the weights based on the results of the original study that is being replicated and the optimality criterion being to maximise the disjunctive power of the trial (the power to reject at least one non-null hypothesis). We show that using the proposed procedure can lead to a substantial increase in the disjunctive power of the replication study, and is robust to changes in the effect sizes between the two studies. Keywords: Familywise Error Rate; Power; Reproducibility; Type I error rate control 1 Introduction Driven by long-standing concerns of a reproducibility crisis in science [ 1 ] , the importance of replicating scientific studies is well-recognised [ 2 ] . Such replication studies are now commonplace in all areas of scientific research. In the context of clinical trials, for example, this has been formalised by the US Food and Drug Administration (FDA) in what is known as the two-trials rule . This states that there should be “at least two pivotal studies, each convincing on its own” [ 3 ] before a drug is approved and “reflects the need for substantiation of experimental results, which has often been referred to as the need for replication of the finding” [ 4 ] . For trials with a single hypothesis, the two-trial rule is usually satisfied by running two independent trials and requiring the results to be statistically significant at the standard (one-sided) 2.5% level [ 5 , 6 ] . In trials (and scientific experiments more generally) that test multiple hypotheses simultaneously, there is the added complication of how to handle this multiplicity. In confirmatory (or ‘pivotal’) trials, where the aim is to provide definitive results, the familywise error rate (FWER) is the typical error rate to be controlled [ 7 , 8 ] . The FWER is defined as the (maximum) probability of falsely rejecting at least one null hypotheses, under any configuration of which hypotheses are null and non-null. Control of the FWER at some small level \alpha is seen as particularly important when the hypotheses are related in some way, such as when testing multiple doses of the same drug or multiple endpoints on the same group of patients. Many multiple testing procedures have been proposed to control the FWER in clinical trials and other scientific contexts more generally [ 9 ] . A popular choice (and perhaps the most common by far) remains the well-known Bonferroni procedure. When testing m hypotheses of interest H_{1},\ldots,H_{m} with corresponding p -values denoted p_{1},\ldots,p_{m} , the Bonferroni procedure rejects hypothesis H_{i} if p_{i}<\alpha/m . A natural extension is to consider a weighted Bonferroni procedure, where instead of ‘splitting’ the allowed type I error ( \alpha ) equally between the hypotheses, pre-determined weights w_{i} are assigned to hypothesis H_{i} (where these weights are non-negative and sum up to 1). Hypothesis H_{i} is then rejected if p_{i}<w_{i}\alpha . Note that setting w_{i}\equiv 1/m recovers the standard (unweighted) Bonferroni procedure. Intuitively, weighting increases the power to reject hypothesis H_{i} when w_{i}>1/m , and decreases the power when w_{i}<1/m , compared with using the standard Bonferroni procedure. These weights could be chosen based on the a-priori relative importance of the hypotheses, or could be chosen to optimise different definitions of the power of the experiment. The latter is the approach we consider in this paper. The idea of choosing the weights to optimise power for Bonferroni procedures has received limited attention in the literature. Almost all of this has focused on maximising either the mean power, the weighted mean power (where the weights reflect their a-priori importance), or the expected number of rejections. In the context of fixed-sequence tests for clinical trials, optimal weights have been provided numerically to maximise the weighted power [ 10 ] and the expected number of rejections [ 11 ] . In three related papers [ 12 , 13 , 14 ] in the context of replication studies for genomics, analytical formulae have been derived for weighted Bonferroni tests to obtain the optimal mean power. As discussed in [ 10 , 15 ] , while the mean (weighted or unweighted) power is easier to optimise numerically, it has issues with interpretability and is not the most common definition of overall power used. For this reason, Xi and Chen [ 15 ] considered choosing the weights to maximise either the disjunctive or conjunctive power of a single trial with multiple hypotheses. The disjunctive power is the probability of rejecting at least one non-null hypothesis, whereas the conjunctive power is the probability of rejecting all non-null hypotheses [ 16 ] . The motivation for using the disjunctive power in the context of a clinical trial would be the desire to find at least one truly effective treatment, or demonstrate the effectiveness of a treatment on at least one of the (co-)primary endpoints of the trial. Xi and Chen [ 15 ] showed how to maximise either the disjunctive or conjunctive power for both independent and correlated test statistics, with the optimal weights found numerically. A key challenge in all of the above literature is that the true optimal weights depend on the true effect sizes (i.e. treatment effects in the clinical trial setting) associated with each hypothesis, which are unknown. In [ 10 , 11 , 15 ] this problem is handled by using a-priori effect sizes for each hypothesis, corresponding to the effect sizes required for the desired marginal power and sample size calculations (for example). Meanwhile, in [ 12 , 13 , 14 ] a sample-splitting approach was proposed, where part of the data from the experiment is used to estimate the true effect size, and the other part of the data is used for hypothesis testing. This latter approach can suffer from a substantial loss in power, however, due to the reduced sample size. In this paper, we take a different approach for the context of replication studies and the two trial rule in particular. Given that an experiment (labelled ’experiment 2’) is designed as a replication study for some earlier experiment (labelled ‘experiment 1’), we can use the data from experiment 1 to estimate the true effect sizes, which are then used to calculate optimal weights for testing the hypotheses in experiment 2. Like in Xi and Chen [ 15 ] , we consider maximising the disjunctive power as our optimality criteria, but propose an alternative numerical solution to

Read the complete text