Investigation of FWER and power for methodological changes introduced in the bdots R package
Collin Nolte · 2025
Read Investigation of FWER and power for methodological changes introduced in the bdots R package by Collin Nolte (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17495.txt.
Introduction
A statistics package used on time-series data had a quiet flaw: it cried significance far too often. Collin Nolte documents the problem and two fixes in a 2025 paper about the BDOTS method, which compares bootstrapped curves over time and lives in an R package.
Simulations across many experimental conditions showed the original method inflating the family-wise error rate well above its promised level. Both proposed alternatives held the rate close to nominal. Nolte also maps how statistical power grows with effect size under each fix, so users can see the price of honesty. The paper closes the loop from discovery to remedy: the flaw found in 2025 is diagnosed, the corrected methods are validated, and the new package version is justified by the evidence shown.
Catalog record
| CREATOR | Collin Nolte |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17495
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigation of FWER and power for methodological changes introduced in the bdots R package Collin Nolte Abstract In 2025, we identified a methodological issue in the bootstrapped differences of times series (BDOTS) first introduced in 2017 resulting in a significant inflation of the family-wise error rate. The goal of the present manuscript is threefold: to identify the problem in the original methodology, to present two alternative solutions, and to compare estimates of the FWER and power of each of the considered methods across a variety of experimental conditions. We find conclusive evidence that the original BDOTS method does inflate the FWER, while each of the proposed alternatives maintain a FWER much closer to the nominal rate. Additionally, we demonstrate the relationship between power and effect size for each of the proposed methods. In total, the results presented justify the methodological changes presented in the new iteration of the bdots package. 1 Introduction A problem ubiquitous throughout the sciences, though in the cognitive sciences especially, is that of statistically analyzing a process unfolding in time. In particular, we consider the problem of comparing a process in time as it evolves differentially between two or more experimental groups. And while there are many techniques for demonstrating that a difference exists, few offer any insight into when . Testing for temporal differences is complicated by the fact that when this process is continuous, there are often an arbitrarily large number of time points that could be compared. This is essentially a problem of multiple comparisons. One approach to this was introduced by Oleson et al. 2017 who presented a modified Bonferroni correction to a series of test statistics, using estimates of autocorrelation between statistics to make the appropriate adjustments to the significance levels to control FWER. This approach, which they named bootstrapped differences in time series, was introduced in the R package bdots [ Seedorff et al. 2018 ] . The modified Bonferonni correction used by bdots relies on the construction of estimated distributions of time series via bootstrapping for each experimental group. These distributions are then used to construct test statistics at each observed time point. A closer look at the original iteration presents concerns as it involves restrictive assumptions on the data that are unlikely to be met in many, if not most situations. This includes data typically collected in the context of the Visual World Paradigm (VWP), a widely used experimental paradigm in language research that involves tracking eye movements from participants in response to spoken language. This is notable in that it was data collected in VWP research that motivated the bdots methodology. There it was maintained that the process of interest in each experimental group assumed a homogeneous mean structure, with no between-subject variability to be accounted for, an assumption we will explore in more detail in the following section. Empirical data collected in a variety of contexts suggests that this assumption is unlikely to be true, the consequence of which is a family-wise error rate that is unacceptably high. This manuscript has three goals: to identify the issue in the original iteration of bdots resulting in an inflated FWER, to propose two solutions to this problem, and to demonstrate the efficacy of the proposed solutions. We begin with a mathematical description of the problem, along with our proposed solutions. These consist of a modification to the original algorithm and the introduction of a permutation test. We then present a collection of simulations evaluating the FWER for each method across a variety of experimental conditions. We conclude with two simulations intended to demonstrate power among the competing methods. Ultimately, the results of these simulations are intended to justify the methodological changes that are now included in the current instantiation of the bdots package [ Nolte et al. 2022 ] . 2 Methods In each of the methods to be described, we begin with the observation of y_{it} for subjects i=1,\dots,n over times t=1,\dots,T . Typically, these subjects fall into different groups g=1,\dots,G , with each group containing n_{g} subjects. We further assume that the empirically observed data derives its mean structure from a parametric function f with associated error: y_{it}=f(t|\theta_{i})+\epsilon_{it} (1) where \theta_{i} is the subject-specific parametrization of f with \epsilon_{it}=\phi\epsilon_{i,t-1}+w_{it},\quad w_{it}\sim N(0,\sigma). (2) Under this paradigm, the errors are permitted to be either iid normal (with \phi=0 ) or have an AR(1) structure, with 0<\phi<1 . It is generally assumed that the observed data across subjects make up a distribution for each group, specified as either a multivariate distribution of the parameters \theta_{i} or a distribution of resulting curves, f(t|\theta_{i}) . We further assume that each subject’s parameters are drawn from a group-level distribution, \theta_{i}\sim N(\mu_{g},V_{g}). (3) Ultimately, it will be from a distribution of curves that we determine the temporal characteristics of each group, with the differences in these temporal characteristics being what we are interested in identifying. With the general notation addressed, we now move to the particulars of each of the methods considered. 2.1 Homogeneous Bootstrap The original bootstrapping algorithm presented in Oleson et al. 2017 follows what we will call the homogeneous means assumption. Accordingly, we will call this bootstrapping algorithm the homogeneous bootstrap . Under the homogeneous means assumption, it is still assumed that observed data for each subject retains the mean structure given in Equation 1 , but with the additional assumption that \theta_{i}=\theta_{j} for all subjects i,j within the same group. In other words, there is assumed to be no variability in the mean structure between subjects within the same group. This is evidenced in the original bootstrapping differences in time series algorithm, which samples without replacement at each bootstrapping step: 1. For each subject, fit a nonlinear regression model to obtain \hat{\theta}_{i} . Oleson et al. 2017 recommends specifying an AR(1) autocorrelation structure for model errors. Assuming large sample normality, the sampling distribution of each estimator can be approximated by a multivariate normal distribution with mean for subject i , \hat{\theta}_{i} corresponding to the point estimate and standard deviations corresponding to the standard errors, s_{i} . 2. Using the approximate sampling distributions in (1), randomly draw one bootstrap estimate for each of the model parameters on every subject \hat{\theta}_{i}^{(b)}\sim N(\hat{\theta}_{i},s_{i}^{2}) (4) 3. Once a bootstrap estimate has been collected for each parameter and for every subject, for each parameter, find the mean of the bootstrap estimates across n_{g} individuals for the b th bootstrap in group g , \theta_{g}^{(b)}=\frac{1}{n_{g}}\sum_{i=1}^{n_{g}}\hat{\theta}_{i}^{(b)} (5) 4. Use the mean parameter estimates to determine a bootstrapped population level curve, which provides the average population response at each time point, f(t|\theta_{g}^{(b)}) . 5. Perform steps (2)-(4) B times to obtain estimates of the population curves. Use these to create estimates of the mean response and standard deviation at each of the time points. For each group g=1,\dots,G , this gives \overline{p}_{gt}=\frac{1}{B}\sum_{b=1}^{B}f(t|\theta_{g}^{(b)}),\qquad s_{gt}^{2}=\frac{1}{B-1}\sum_{b=1}^{B}[f(t|\theta_{g}^{(b)})-\overline{p}_{gt}]^{2}, (6) where \overline{p}_{gt} and s_{gt}^{2} are mean and variance estimates at each time point for group g . Population means and standard deviations at each time point for each of the groups were used to construct a series of (correlated) test statistics, where the family-wise error rate was controlled by using the modified Bonferonni correction introduced in Oleson et al. 2017 to test for