Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R.
Kumle L, Võ ML, Draschkow D. · 2021
Read Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R. by Kumle L, Võ ML, Draschkow D. (2021) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc8613146.txt.
Introduction
Kumle, Võ, and Draschkow give a hands-on guide to power checks for mixed-effects models. These models handle fixed and random effects at once. No simple formula gives the chance that a test will find a real effect. That chance, called power, shapes sample size choices and trust in results.
The tutorial uses simulation to estimate power. It covers three cases: planning from past data, choosing how many test items to use, and planning with no past data. Examples span plain and generalized models in R. Code and open datasets come with the guide. The aim is sound study design for researchers who use these models.
Catalog record
| CREATOR | Kumle L, Võ ML, Draschkow D. |
|---|---|
| YEAR | 2021 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Estimating power in (generalized) linear mixed models: An open introduction and tutorial in R By Leah Kumle, Melissa L-H Võ, Dejan Draschkow Published in: Behavior research methods DOI: 10.3758/s13428-021-01546-0 ABSTRACT Abstract Mixed-effects models are a powerful tool for modeling fixed and random effects simultaneously, but do not offer a feasible analytic solution for estimating the probability that a test correctly rejects the null hypothesis. Being able to estimate this probability, however, is critical for sample size planning, as power is closely linked to the reliability and replicability of empirical findings. A flexible and very intuitive alternative to analytic power solutions are simulation-based power analyses. Although various tools for conducting simulation-based power analyses for mixed-effects models are available, there is lack of guidance on how to appropriately use them. In this tutorial, we discuss how to estimate power for mixed-effects models in different use cases: first, how to use models that were fit on available (e.g. published) data to determine sample size; second, how to determine the number of stimuli required for sufficient power; and finally, how to conduct sample size planning without available data. Our examples cover both linear and generalized linear models and we provide code and resources for performing simulation-based power analyses on openly accessible data sets. The present work therefore helps researchers to navigate sound research design when using mixed-effects models, by summarizing resources, collating available knowledge, providing solutions and tools, and applying them to real-world problems in sample sizing planning when sophisticated analysis procedures like mixed-effects models are outlined as inferential procedures. Keywords: Power, Mixed models, Simulation, lme4, mixedpower, R INTRODUCTION Linear mixed-effects models (LMMs), as well as generalized linear mixed models (GLMMs), are a popular and powerful choice in cognitive research, as they allow between-subject and between-item variance to be estimated simultaneously (for a discussion see Baayen, Davidson, & Bates, 2008; Kliegl, Wei, Dambacher, Yan, & Zhou, 2011). Moreover, (G)LMMs offer flexibility in dealing with missing data and unbalanced designs, and allow for a unified treatment of continuous and categorical responses (Baayen et al., 2008). There are plenty of excellent resources aimed at using (G)LMMs (e.g. Baayen et al., 2008; Bates, Mächler, Bolker, & Walker, 2014; DeBruine & Barr, 2021; Goldstein, 2007; Harrison et al., 2018), but far less guidance is available for experimental psychologists who want to estimate power for (G)LMMs (Brysbaert & Stevens, 2018; Westfall, Kenny, & Judd, 2014). However, accounting for statistical power while planning experimental designs is important for the reliability and replicability of empirical findings and a critical step for the successful preregistration of studies. In this tutorial, we will consider different scenarios for a simulation-based power analysis and provide examples on how to perform such an analysis. This paper is designed to meet the needs of researchers who have some experience with mixed-effects modeling, and thus does not discuss topics such as model selection or optimal random-effects structure (Barr, Levy, Scheepers, & Tily, 2013; Bates, Kliegl, Vasishth, & Baayen, 2015a). Consequently, all scenarios introduced in this tutorial assume that an optimal model has already been selected. Thus, our aim is to provide tools and resources for researchers to use and explore simulation-based procedures, empowering them to find solutions for their own specific use cases. To achieve this, we use real-world data sets, rather than simplified examples, hoping that these examples would cover a wide problem space. Why is power important? Statistical power is defined as the probability of rejecting the null hypothesis in favor of an alternative hypothesis when the null hypothesis indeed is false (Johnson, Barry, Ferguson, & Müller, 2015; Maxwell, Kelley, & Rausch, 2008). It is expressed by (1 – β), where β is the type II error probability—in simple terms, “if there is an effect of a certain size, what is the probability that my study will detect it”. The level of statistical power may be determined by balancing the researcher’s goal with the effort/cost needed to further increase power (Brysbaert & Stevens, 2018; O’Brien & Castelloe, 2007); however, in cognitive sciences, “80%” is often used as a minimal value. Statistical power for any experimental design depends predominantly on sample size, number of items (or trials), effect size, measurement variability, and the number of comparisons being performed (Coppock, 2013; Gelman & Carlin, 2014) (for a more detailed introduction to statistical power see Cohen, 1988). From these factors, sample size or number of items (i.e. number of observations) can be manipulated most easily. Keeping in mind the superordinate goal of conducting adequately powered studies, a power analysis is a helpful tool for planning your sample size (Nakagawa & Foster, 2004), which justifies the need for appropriate tools and guidelines to conduct such power analyses when sophisticated analysis procedures like (G)LMMs are used as inferential procedures. Why is it difficult to estimate power for (G)LMMs? While (G)LMMs offer many advantages over traditional analysis procedures (e.g. conventional linear models, ANOVAs), their use is not as straightforward and requires careful deliberation (Matuschek, Kliegl, Vasishth, Baayen, & Bates, 2017). In addition to the specification of main effects and their interactions (i.e. fixed effects), (G)LMMs allow for the specification of parameters associated with the variance and correlation of random factors (e.g. of subjects and items) (Matuschek et al., 2017). As noted earlier, the power of a specific design is also influenced by the variability of responses. Consequently, all factors that have an influence on the variability of responses need to be accounted for when estimating power (Westfall et al., 2014). Since (G)LMMs capture multiple sources of random variations (Westfall et al., 2014), suitable power analyses procedures need to appropriately account for this increased complexity. Therefore, the same aspects that lead to the advantages of (G)LMMs and make them a popular tool in cognitive research also lead to increased difficulties in model specification and the analysis of power. Classical approaches to power analysis typically work with analytical formulas which lack the necessary flexibility (Johnson et al., 2015; Thomas & Juanes, 1996) to solve power estimations for (G)LMMs (Brysbaert & Stevens, 2018; Green & MacLeod, 2016; Thomas & Juanes, 1996). One notable exception is Westfall, Kenny, and Judd’s (2014) analytical solution to power for mixed models with crossed random effects (e.g. a sample of participants respond to a sample of stimuli) (Shinyapp: https://jakewestfall.shinyapps.io/crossedpower/). Although extremely useful in certain cases, this solution is only applicable when calculating power for models with one fixed effect with two levels (Brysbaert & Stevens, 2018), i.e. very simple models. As more complex models are often used in practice, and power estimates for more than one fixed effect per model therefore become more frequent, the need for a different approach for (G)LMM power analysis becomes apparent. A flexible and very intuitive alternative to analytic power solutions are simulation-based power analyses (Brysbaert & Stevens, 2018; Thomas & Juanes, 1996). How does simulation-based power estimation work? In simple terms, one basic question behind power analyses is: “Suppose there really is an effect of a certain size and I run my experiment one hundred times-how many times will I get a statistically significant result?” (Coppock, 2013). As it is possible to simulate the outcome of an experiment, power can be calculated based on the proportion of significant simulations to all simulations (Johnson et