An information metric for comparing and assessing informative interim decisions in sequential clinical trials
Gianmarco Caruso, Wlliam F. Rosenberger, Pavel Mozgunov, Nancy Flournoy · 2025
Read An information metric for comparing and assessing informative interim decisions in sequential clinical trials by Gianmarco Caruso, Wlliam F. Rosenberger, Pavel Mozgunov, Nancy Flournoy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04904.txt.
Introduction
Stopping a clinical trial early can distort the final estimate of a treatment's effect. G. Caruso, lead author with three co-authors, propose an information metric for that distortion in a 2025 statistics paper. Group sequential designs allow interim looks and early stopping for benefit or futility, but ignoring those decisions can bias posterior estimates.
Using Kullback-Leibler divergence, a measure of difference between distributions, they quantify distortion across the whole parameter space. Comparing decision boundaries and priors shows how the metric can guide future trial planning. An expected version helps clinicians choose boundaries, adding to standard error-rate control.
Catalog record
| CREATOR | Gianmarco Caruso, Wlliam F. Rosenberger, Pavel Mozgunov, Nancy Flournoy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04904
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An information metric for comparing and assessing informative interim decisions in sequential clinical trials Gianmarco Caruso, Wlliam F. Rosenberger, Pavel Mozgunov, Nancy Flournoy Abstract Group sequential designs enable interim analyses and potential early stopping for efficacy or futility. While these adaptations improve trial efficiency and ethical considerations, they also introduce bias into the adapted analyses. We demonstrate how failing to account for informative interim decisions in the analysis can substantially affect posterior estimates of the treatment effect, often resulting in overly optimistic credible intervals aligned with the stopping decision. Drawing on information theory, we use the Kullback-Leibler divergence to quantify this distortion and highlight its use for post-hoc evaluation of informative interim decisions, with a focus on end-of-study inference. Unlike pointwise comparisons, this measure provides an integrated summary of this distortion on the whole parameter space. By comparing alternative decision boundaries and prior specifications, we illustrate how this measure can improve the understanding of trial results and inform the planning of future adaptive studies. We also introduce an expected version of this metric to support clinicians in choosing decision boundaries. This guidance complements traditional strategies based on type-I error rate control by offering insights into the distortion introduced to the treatment effect at each interim phase. The use of this pre-experimental measure is finally illustrated in a group sequential trial for evaluating a treatment for central nervous system disorders. Keywords adaptive clinical trials \cdot group sequential designs \cdot Kullback-Leibler divergence \cdot sequential experiment 1 Introduction Group sequential designs (GSDs) extend traditional fixed-sample designs (FSDs) by incorporating one or more interim analyses, allowing for periodic evaluation of accumulating data in clinical trials ( Jennison and Turnbull, 1999 ) . This approach enables early termination of the trial for efficacy, when strong evidence supports the new treatment, or for futility, reducing patient exposure to ineffective interventions. By adapting to emerging results, group sequential designs can reduce the size and duration of the trial, potentially accelerating the approval of effective therapies. In recent years, GSDs have gained popularity, driven by a call for more efficient and timely evaluation of interventions. Notably, the COVID-19 pandemic underscored the need for adaptive trial methodologies capable of delivering rapid results without compromising scientific integrity ( Stallard et al., 2020a ; Kunz et al., 2020 ; Kunz et al., 2024 ) . The increased flexibility offered by GSDs comes at the cost of greater complexities in preserving the desired type-I error probability, defined in this context as the probability of incorrectly rejecting the null hypothesis at any interim look. In fact, using the same critical value at each interim look as in a FSD leads to an inflation of the overall type-I error rate because of the repeated testing ( Armitage et al., 1969 ) . Various types of critical values, also referred to as stopping boundaries, have been proposed in the literature to ensure type-I error rate control in sequential settings. Notably, Pocock, 1977 introduced an approach that uses constant decision boundaries across all interim analyses, resulting in a relatively high probability of rejecting at an early interim analysis. O’Brien and Fleming, 1979 proposed boundaries that are more conservative in early analyses and become less stringent in later stages, making it more difficult to reject the null hypothesis at early analyses. Although the type-I error rate is traditionally associated with the frequentist framework, methods to control it are often employed in association with Bayesian approaches ( Gsponer et al., 2014 ) . While much of the methodological development in adaptive designs has focused on defining boundaries to control type-I error rates and to ensure adequate power, less attention has been given to addressing the impact that interim looks have on the estimation of treatment effects and the associated uncertainty. For example, trials with promising early results are more likely to proceed, while those with weaker effects may be stopped early for futility. This creates a bias in the estimation of the treatment effect, as early stopping tends to occur when interim results show an overestimated treatment benefit ( Freedman and Spiegelhalter, 1989 ) . As a consequence, final estimates of the treatment effect can be systematically inflated, and confidence or credible intervals may not reflect the true uncertainty or may not maintain the correct coverage. Flournoy and Oron, 2020 show that the bias of cumulative sample means is proportional to minus the covariance between the sum of observed outcomes and the stopping decision, which determines the sample size. For a methodological review of the approaches proposed to mitigate this issue in point and interval estimation, the reader is referred to Jennison and Turnbull, 1990 ; Jennison and Turnbull, 1993 and Robertson et al., 2023 ; Robertson et al., 2024 . Several studies have investigated the impact of informative interim decisions on the estimation of the treatment effect and the associated loss of information. In particular, Marschner, 2021 introduces a framework that decomposes the full likelihood into the design likelihood and the conditional-on-decisions likelihood, providing a meta-analytic perspective to assess estimation bias. Tarima and Flournoy, 2024 examine how informative interim adaptations influence the distribution of sample-based statistics, defining the “cost of adaptation" as the proportion of Fisher information consumed by these adaptations. Posterior distributions, which encapsulate all available knowledge and uncertainty about the parameter of interest after observing the data, serve as the core of all Bayesian posterior estimates and probability statements ( Gelman et al., 1995 ) . In this respect, Flournoy and Tarima, 2023 advocate constructing Bayesian posteriors that incorporate the informative interim decisions made during a group sequential trial. This approach treats these decisions as part of the observed data, thus capturing all available information more accurately than traditional posteriors based solely on accumulated responses ( Roberts, 1967 ) . Here, informative stopping rules refer to decisions taken at pre-planned interim analyses on the basis of accumulating data related to the primary outcome ( Berger and Wolpert, 1988 , p. 88) . Building on this, we extend the concept of conditioning on the realized design to multi-stage settings and introduce a novel metric to quantify how interim analyses in a GSD modify posterior beliefs about the treatment effect compared to a FSD. As we will show, the same data can yield very different posterior estimates depending on whether informative adaptation is taken into account in the analysis, making it crucial to have a global measure of this discrepancy. This discrepancy is relevant because, in many applied analyses, posteriors are obtained without incorporating informative adaptive rules, thus overlooking that such rules can themselves convey information about the parameter ( Flournoy and Tarima, 2023 ) . The proposed metric is derived from the information-theoretic concept of entropy, which has seen wide application in Bayesian statistics, for example, to improve robust inference ( Hooker and Vidyashankar, 2014 ; Miller and Dunson, 2018 ) or optimize resource allocation in experimental design ( Mozgunov and Jaki, 2019 ; Mozgunov and Jaki, 2020 ; Caruso and Mozgunov, 2024 ; Rainforth et al., 2024 ) . Since the realized sequence of decisions (e.g., whether to stop or continue at each interim look) depends on the observed data, the natural use of this metric is for post-hoc evaluation, quantifying how ignoring information from