Power Analysis for Prediction-Powered Inference
Yiqun T. Chen 1,2 , Moran Guo 1 , and Shengyi Li 1 1 Department of Biostatistics, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. 2 Department of Computer Science, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. [email protected] · 2026
Read Power Analysis for Prediction-Powered Inference by Yiqun T. Chen 1,2 , Moran Guo 1 , and Shengyi Li 1 1 Department of Biostatistics, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. 2 Department of Computer Science, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16041.txt.
Introduction
Chen, Guo, and Li size studies that lean on machine-predicted outcomes. Prediction-powered inference gives valid tests from partly labeled data. Classic power formulas ignore the predicted half of the data. Their work asks how many labels a set power truly needs. Closed forms follow from the estimator's large-sample variance. Wald test inversion turns variance into required label counts. Two-sample tests and 2x2 risk tables both get formulas. Label savings scale with prediction R squared, a handy rule. Three biomedical cases show the framework at work.
Catalog record
| CREATOR | Yiqun T. Chen 1,2 , Moran Guo 1 , and Shengyi Li 1 1 Department of Biostatistics, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. 2 Department of Computer Science, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78847 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16041
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Power Analysis for Prediction-Powered Inference Yiqun T. Chen 1,2 , Moran Guo 1 , and Shengyi Li 1 1 Department of Biostatistics, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. 2 Department of Computer Science, Johns Hopkins University, Baltimore, Maryland 21205, U.S.A. [email protected] Abstract Modern studies increasingly leverage outcomes predicted by machine learning and artificial intelligence (AI/ML) models, and recent work, such as prediction-powered inference (PPI), has developed valid downstream statistical inference procedures. However, classical power and sample size formulas do not readily account for these predictions. In this work, we tackle a simple yet practical question: given a new AI/ML model with high predictive power, how many labeled samples are needed to achieve a desired level of statistical power? We derive closed-form power formulas by characterizing the asymptotic variance of the PPI estimator and applying Wald test inversion to obtain the required labeled sample size. Our results cover widely used settings including two-sample comparisons and risk measures in 2 × 2 2\times 2 tables. We find that a useful rule of thumb is that the reduction in required labeled samples relative to classical designs scales roughly with the R 2 R^{2} between the predictions and the ground truth. Our analytical formulas are validated using Monte Carlo simulations, and we illustrate the framework in three contemporary biomedical applications spanning single-cell transcriptomics, clinical blood pressure measurement, and dermoscopy imaging. We provide our software as an R package and online calculators at https://github.com/yiqunchen/pppower . Keywords: Artificial intelligence; Experimental design; Label efficiency; Power analysis; Prediction-powered inference; Sample size. 1 Introduction Gold-standard labels are fundamental for scientific discoveries but remain time-consuming and expensive: Annotating cell types in single-cell RNA-seq requires expert curation of marker genes ( 3 ) ; measuring clinical outcomes in randomized trials demands patient follow-up and laboratory assays ( 20 ) ; and evaluating the quality of large language model (LLM) responses to complex questions needs human expert judgments ( 8 ) . At the same time, predictions from machine learning (ML) and artificial intelligence (AI) models have become more accurate and available, where millions of cells can be labeled in minutes, prognostic models can score electronic health records at negligible cost, and “LLM-as-a-judge” pipelines, where an LLM serves as the human expert, can evaluate millions of responses in minutes. With the rise of these models, it is natural to ask whether these predictions can reduce the number of costly labels needed to answer scientific questions of interest, e.g., what is the prevalence of this disease or the change in cell-type proportions across different experimental conditions. Recent work has shown that treating predicted labels as direct observations is invalid and inflates Type I error. Methods such as Prediction-Powered Inference (PPI) ( 1 ) and its tuned variant PPI++ ( 2 ) instead calibrate predictions against a small set of gold-standard labels. Together with related work ( 26 ; 17 ; 11 ; 14 ) , these methods yield unbiased and asymptotically more efficient estimators than classical analyses based on gold-standard labels alone. However, less work has translated those efficiency gains into the design-stage question every study planner must answer: Given a predictive model’s accuracy, how many labeled samples are needed to detect an effect of interest with the desired statistical power? Classical power formulas ( 12 ; 13 ) do not provide a straightforward way to distinguish between gold-standard labels and predictions. As a result, practitioners either ignore predictions entirely, which leads to valid planning but potential losses in efficiency, or naively treat predictions as gold-standard labels, which leads to overestimated power and underestimated sample sizes. Several recent contributions start to address this gap: 2 develop cost-allocation routines from pilot data but do not provide closed-form power or sample-size formulas. Comprehensive software packages ( 22 ; 10 ) unify a range of post-prediction inference methods yet focus on the analysis stage and offer no prospective study-design functionality. 18 derive power reductions for ANCOVA-style regression adjustment but restrict attention to that single estimator. While most work focused on comparing asymptotic efficiency of estimators, 16 also investigated the finite-sample trade-off, and our rule of thumb gives that limitation a concrete planning interpretation. Closest to our work is 4 , who invert the PPI++ variance formula to jointly optimize the labeled and unlabeled sample sizes under a cost constraint. Our starting point differs in a practical but important way: in many biomedical settings the two sample sizes are not jointly optimized. The pool of unlabeled observations is often fixed by the experiment itself (e.g., the number of cells captured in a single-cell assay, or the patients meeting inclusion criteria), and the binding constraint is how many a biologist or physician can manually label. We therefore condition on a given N and ask how many gold-standard labels n are needed to achieve target power. In this paper, we propose pppower , a framework and R package for prediction-powered power analysis (see Figure 1 ). Our contributions are as follows: 1. Closed-form formulas. We derive power and sample-size formulas for prediction-powered estimators that combine a smaller set of gold-standard labels with a larger pool of model-based predicted labels, and show that they reduce to the classical formulas when the predictions are uninformative. These formulas cover two-sample comparisons, paired designs, odds ratios and relative risks in 2\times 2 tables, and regression contrasts for (generalized) linear models (Propositions 1 – 6 , 7 , and 8 ). A simple rule of thumb emerges: the required labeled sample size drops by approximately R^{2}\times 100\% . 2. User-friendly parameterization. The formulas accept R^{2} , sensitivity/specificity, or confusion matrices as inputs for the one-sample and binary-table designs, avoiding direct covariance specification in the main operational cases; regression-contrast extensions still require contrast-level pilot covariance blocks (Sections 3.3 and 4.4 ). 3. Simulation validation. A comprehensive simulation study validates all formulas against Monte Carlo estimates and establishes practical guidance on when model-based predicted labels yield the largest reductions in required labeled sample size (Section 5 ). 4. Open-source software with case study. An R package with a pwr -style API and an accompanying online calculator unify power calculation and sample-size determination in a single function call. We demonstrate the package on three biomedical case studies (Section 6 ). Y : true labels, f : predictions, n : paired, N : predictions only Estimator \hat{\theta}_{\lambda}=\bar{Y}_{n}+\lambda(\bar{\tilde{f}}_{N}-\bar{f}_{n}) Power and sample size invert \mathrm{Var}(\hat{\theta}_{\lambda}) to obtain required labels n^{\star} (a) Estimator-and-inversion schematic. (b) NHANES planning by surrogate model. Figure 1 : Prediction-powered planning. Panel (a) shows the notation, estimator, and variance inversion. Panel (b) shows the NHANES sample-size plan for \Delta=4 mmHg: gray is the classical design, and the orange curves are the age-only linear model, the richer clinical linear model, and the random forest surrogate. 2 Setup and Background We consider i.i.d. labeled data \{(X_{i},Y_{i})\}_{i=1}^{n} drawn from a joint distribution P_{XY} on \mathcal{X}\times\mathcal{Y} , and an independent pool of unlabeled covariates \{\widetilde{X}_{j}\}_{j=1}^{N} drawn from the same marginal P_{X} , with r=n/N denoting the labeled-to-unlabeled ratio. Throughout, n denotes the number of gold-standard labels and N the size of the larger unlabeled sample carrying model-based predictions, with the practically important regime often having N\gg n . A predictor f:\mathcal{X}\to\mathcal{Y} yields predictions f_{i}=f(X_{i}) on labeled observations and \widetilde{f}_{j}=f(\widetilde{X}_{j}) on