Combined P -value Functions for Compatible Effect Estimation and Hypothesis Testing in Drug Regulation
Samuel Pawel, Małgorzata Roos, Leonhard Held · 2025
Read Combined P -value Functions for Compatible Effect Estimation and Hypothesis Testing in Drug Regulation by Samuel Pawel, Małgorzata Roos, Leonhard Held (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10246.txt.
Introduction
Samuel Pawel, Małgorzata Roos, and Leonhard Held addressed a quiet inconsistency in drug regulation in 2025. Regulators require two significant trials to approve a drug, the two-trials rule. But when officials then estimate the drug's effect, they usually pool both trials in a fixed-effect meta-analysis. That pooled interval can exclude no-effect even when the two-trials rule itself was not satisfied, so testing and estimation can contradict each other.
The authors recast both procedures in one framework of combined p-value functions. In that view, the two-trials rule matches one classical combination method and meta-analysis matches another. From the unified frame they derive compatible p-values, effect estimates, and confidence intervals in closed form. Compatibility means a regulator never has to announce significance with one hand and take it back with the other. The paper also offers new proposals for combining evidence coherently.
Catalog record
| CREATOR | Samuel Pawel, Małgorzata Roos, Leonhard Held |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10246
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Combined P -value Functions for Compatible Effect Estimation and Hypothesis Testing in Drug Regulation Samuel Pawel, Małgorzata Roos, Leonhard Held Abstract The two-trials rule in drug regulation requires statistically significant results from two pivotal trials to demonstrate efficacy. However, it is unclear how the effect estimates from both trials should be combined to quantify the drug effect. Fixed-effect meta-analysis is commonly used but may yield confidence intervals that exclude the value of no effect even when the two-trials rule is not fulfilled. We systematically address this by recasting the two-trials rule and meta-analysis in a unified framework of combined p -value functions, where they are variants of Wilkinson’s and Stouffer’s combination methods, respectively. This allows us to obtain compatible combined p -values, effect estimates, and confidence intervals, which we derive in closed-form. Additionally, we provide new results for Edgington’s, Fisher’s, Pearson’s, and Tippett’s p -value combination methods. When both trials have the same true effect, all methods can consistently estimate it, although some show bias. When true effects differ, the two-trials rule and Pearson’s method are conservative (converging to the less extreme effect), Fisher’s and Tippett’s methods are anti-conservative (converging to the more extreme effect), and Edgington’s method and meta-analysis are balanced (converging to a weighted average). Notably, Edgington’s confidence intervals asymptotically always include the individual trial effects, while meta-analytic confidence intervals shrink to a point at the weighted average effect. We conclude that all of these methods may be appropriate depending on the estimand of interest. We implement combined p -value function inference for two trials in the R package twotrials , allowing researchers to easily perform compatible hypothesis testing and effect estimation. Publication type: Research Article keywords Confidence interval, estimand, median estimate, meta-analysis, two-trials rule † † titlemark: Combined P -value Functions for Compatible Effect Estimation and Hypothesis Testing in Drug Regulation † † corresponding: Samuel Pawel, Epidemiology, Biostatistics and Prevention Institute, Hirschengraben 84, 8001, Zurich, Switzerland. Pawel et al. 1 Introduction The “two-trials rule” in drug regulation requires “ at least two adequate and well-controlled studies, each convincing on its own ” for the demonstration of drug efficacy and subsequent market approval ( 1 , p.3) . This criterion reflects the need for “substantiation” and “replication” of scientific results ( 2 , p.8) , and is typically implemented by requiring the p -values from the two trials to be statistically significant at the conventional (one-sided) \alpha=0.025 level. However, this procedure alone does not provide a combined effect estimate nor a confidence interval (CI), and it has been suggested to pool the estimates with fixed-effect meta-analysis for this purpose ( 3 , 4 , 5 ) . Yet, the meta-analytic CI and point estimate are not always compatible with the two-trials rule. The meta-analytic CI may exclude the null value while the two-trials rule is not fulfilled, leading to discrepancies that are difficult to interpret and communicate. The results from the two RESPIRE trials ( 6 , 7 , 8 ) in Table 1 illustrate this phenomenon. While the p -value for the null hypothesis of no effect from RESPIRE 1 is p=0.004<0.025 , the p -value from RESPIRE 2 is p=0.144>0.025 . Hence, the two-trials rule is not fulfilled at \alpha=0.025 . At the same time, the 95% CI for the log rate ratio based on combining the trials’ log rate ratio effect estimates with fixed-effect meta-analysis ranges from -0.58 to -0.08 and thus excludes the value of 0. Table 1: Results from the RESPIRE trials regarding the effect of ciprofloxacin after 14 days for the treatment of non-cystic fibrosis bronchiectasis ( 6 , 7 , 8 ) . Log rate ratio Confidence interval (95%) P -value (one-sided) RESPIRE 1 -0.49 -0.85 to -0.13 0.004 RESPIRE 2 -0.18 -0.53 to \phantom{-}0.16 0.144 Meta-analysis -0.33 -0.58 to -0.08 0.004 A first attempt at resolving the apparent paradox could be to realize that the confidence level of the CI does not align with the level of the implicit test underlying the two-trials rule. Since the two-trials rule decision is based on two independent tests at level \alpha=0.025 , the overall test is at level \alpha^{2}=0.000625 , thus one could instead take a (1-2\,\alpha^{2})\times 100\%=99.875\% meta-analytic CI ( 9 , 10 ) . For the RESPIRE trials, this would lead to a meta-analytic 99.875\% CI from -0.71 to 0.05 which includes the value of 0 and hence aligns with the two-trials rule decision. However, the level \alpha=0.025 is arbitrary and it would be desirable to have a CI that is compatible with the two-trials rule for any level, which is still not the case. For example, for \alpha=0.05 , the two-trials rule is still not fulfilled, while the (1-2\,\alpha^{2})\times 100\%=99.5\% meta-analytic CI from -0.66 to -0.01 excludes zero. Despite the widespread use of the two-trials rule in regulatory decision-making ( 11 ) , it remains unclear how point and interval estimation should be reconciled with it. This paper aims to resolve this issue with a new approach. The key idea is to look at both the two-trials rule and meta-analysis from the perspective of p -value functions ( 12 , 13 , 14 , 15 , 16 ) and p -value combination methods ( 17 , 18 , 19 , 20 , 21 ) . The two-trials rule can be understood as a combined p -value function based on the squared maximum of two p -values ( 22 ) which is a special case of Wilkinson’s combination method ( 23 ) , while meta-analysis corresponds to the combined p -value function based on Stouffer’s p -value combination method ( 24 ) with suitable weights. Both can be used to obtain combined p -values for the null hypothesis of no effect, CIs, and point estimates. These quantities are compatible in the sense that the (two-sided) p -value for a null value is less than \alpha if and only if the null value is excluded by the (1-\alpha)\times 100\% CI, and that the point estimate is included in the CI at any confidence level (1-\alpha)\in(0,1) . However, as we will show, the two methods implicitly target different estimands, which explains their different behaviors, and highlights the need to choose the method depending on the scientific question and corresponding estimand of interest. Moreover, the combined p -value function pespective suggests considering alternative p -value combination methods, for example, Edgington’s method based on the sum of p -values ( 25 ) or Fisher’s method based on the product of p -values ( 26 ) . All these p -value combination methods have been studied before in terms of hypothesis testing properties, such as admissibility or monotonicity ( 27 , 17 ) . In this paper, we take an alternative estimation perspective motivated by practical issues in drug regulation. This paper is organized as follows: We begin by summarizing the general theory of combined p -value functions (Section 2 ), followed by investigating combined p -value functions based on the two-trials rule (Section 2.1 ), meta-analysis (Section 2.2 ), Tippett’s method (Section 3.1 ), Fisher’s and Pearson’s methods (Section 3.2 ), and Edgington’s method (Section 3.3 ) in more detail. For each, we derive corresponding point and interval estimates and investigate their properties. Results from