Asymptotic Standard Errors for Reliability Coefficients in Item Response Theory

2025

Read Asymptotic Standard Errors for Reliability Coefficients in Item Response Theory (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22924.txt.

Introduction

Youjin Sung and Yang Liu derive standard errors for reliability coefficients under item response theory, in a 2025 paper. Reliability tells how much trust to place in a test score. Recent reviews split reliability measures into two families. Both families need honest measures of their own sampling error. Existing formulas cover only part of that error.

Test scores wobble from two sources at once. Item parameters are estimated with error, and population moments get replaced by sample ones. The authors give a general strategy that counts both sources together. Worked cases show how to apply the formulas. Test developers can then report reliability with proper uncertainty. That matters for high-stakes exams and clinical scales alike.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22924

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Asymptotic Standard Errors for Reliability Coefficients in Item Response Theory Abstract In a recent review, Liu, Pek, & Maydeu-Olivares (2025b) classified reliability coefficients into two types: classical test theory (CTT) reliability and proportional reduction in mean squared error (PRMSE). This article focuses on quantifying the sampling variability of these coefficients under item response theory (IRT) models. While some existing standard error (SE) formulas are accurate when variability arises only from item parameter estimation, the reliability estimators considered in our work involve additional variability from substituting population moments with sample moments. We propose a general strategy to derive SEs that incorporates both sources of sampling error simultaneously, enabling the estimation of model-based reliability coefficients and their SEs in such settings . We then apply our general theory to derive SEs for two specific estimators under the graded response model: (1) CTT reliability for the expected a posteriori score of the latent variable and (2) PRMSE for the latent variable. Simulation results show that the derived SEs accurately capture the sampling variability across various test lengths in moderate to large samples. We conclude with an empirical illustration and directions for future research. IRT Reliability \authorsnames Youjin Sung, Yang Liu \authorsaffiliations Department of Human Development and Quantitative Methodology, University of Maryland \authornote Correspondence should be made to Youjin Sung at 1232 Benjamin Bldg, 3942 Campus Dr, University of Maryland, College Park, MD, 20742. Email: [email protected]. keywords reliability, item response theory, asymptotic standard errors 1 Introduction Reliability is an overall index of measurement precision ( American Educational Research Association et al., 2014a ) . Given a latent variable (LV) measurement model, a reliability coefficient quantifies how well observed scores 1 1 1 Throughout the paper, the term “observed scores” refers to scores that are functions of response variables, encompassing not only summed scores but also factor scores that depend on model parameters. This interpretation aligns with Liu et al. (2025a, 2025b). , which are functions of response variables, align with latent scores, which are functions of LVs that reflect constructs of interest ( Liu et al., 2025b ) . Recently, Liu et al. (2025c) introduced a regression-based framework of measurement precision, which defines reliability as coefficients of determination associated with regressions. Within this framework, classical test theory (CTT) reliability corresponds to the coefficient of determination when regressing an observed score onto all LVs in the measurement model. Meanwhile, proportional reduction in mean squared error (PRMSE; Haberman & Sinharay, 2010a ) , another popular index of measurement precision in item response theory (IRT; Thissen & Steinberg, 2009a ) , corresponds to the coefficient of determination when regressing a latent score onto all response variables. 2 2 2 Liu et al. (2025c) reserved the term “reliability” exclusively for CTT reliability. To be more consistent with the IRT literature (e.g. Haberman & Sinharay, 2010a ; Liu et al., 2025b ) , we treat both CTT reliability and PRMSE as reliability coefficients in this article. While Liu et al. ’s ( 2025c ) regression formulation of CTT reliability and PRMSE provides a conceptual framework for understanding reliability, in practice, reliability coefficients are estimated from sample data and thus are subject to sampling variability. This inherent uncertainty should be quantified and communicated through standard errors (SEs) and confidence intervals (CIs; e.g., Fan & Thompson, 2001a ) ; however, existing literature on this topic is sparse. The primary goal of this paper is to address this gap by analytically deriving asymptotic SEs for CTT reliability and PRMSE, assuming an IRT model as the underlying LV measurement model. So far, only a few studies have focused on computing SEs or CIs for reliability coefficients within the IRT framework. Andersson & Xin (2018a) derived asymptotic SEs for the so-called marginal reliability ( Cheng et al., 2012a ) and test reliability ( Kim & Feldt, 2010a ) using the Delta method, both of which can be viewed as forms of CTT reliability for specific observed scores, as discussed in later sections. Alternatively, Yang et al. (2012a) obtained SEs for the marginal reliability based on multiple imputation. These approaches provide valid uncertainty quantification when the reliability coefficient can be expressed solely as a function of item parameters; namely, when sampling variability arises only from item parameter estimation. However, the formulas of CTT reliability and PRMSE involve population moments (e.g., variances) implied by the measurement model (Liu et al., 2025a, 2025b). Computing these population moments, given estimated item parameters, is feasible for relatively short tests, but quickly becomes computationally infeasible as test length increases because the total number of possible response patterns grows exponentially. Although certain population variances can be approximated via increasing-test-length asymptotics (e.g., for marginal reliability), such approximations are typically tied to specific types of scores (e.g., maximum likelihood scores) and are useful only under limited circumstances (e.g., when the test is very long). Consequently, when reliability is viewed more generally through the regression framework and the test involves many items, a preferable alternative is to substitute population moments with sample estimates (e.g., empirical reliability; Chalmers, 2012a ) . To provide valid uncertainty quantification in such scenarios, this study presents a general SE derivation for reliability coefficients that are subject to two sources of sampling variability: item parameter estimation and the use of sample moments. In theory, our method is applicable to CTT reliability for any observed score and PRMSE for any latent score, under any fully specified parametric IRT model. We provide full derivations for two specific reliability coefficients as examples in the present work: (1) CTT reliability for the expected a posteriori (EAP) score of the LV and (2) PRMSE for the LV. The derivations are provided under the unidimensional graded response model (GRM; Samejima, 1969a ) for polytomous items, which encompasses the two-parameter logistic (2PL) model ( Birnbaum, 1968a ) as a special case for dichotomous items. While not explicitly addressed here, the proposed derivation readily extends to other observed or latent scores under other types of parametric IRT models. The remainder of the paper is structured as follows. We begin by introducing IRT and provide a brief review of the regression framework of reliability. We then summarize how existing IRT reliability coefficients can be classified under the regression framework. After reviewing the literature of SE estimation for reliability coefficients, we present the general theoretical framework for deriving SEs and apply it to the two example reliability coefficients under the GRM. The finite sample performance of the derived SEs is then evaluated in a simulation study for both the 2PL model and the GRM, followed by an empirical illustration. Finally, we conclude with a summary and a discussion of the limitations and potential extensions of this study. 2 IRT Reliability from a Regression Framework 2.1 Item Response Theory Let \Theta_{i} denote a unidimensional LV for person i , which is assumed to follow a standard normal distribution. Let Y_{ij} denote a random response variable for person i on item j and \mathbf{Y}_{i}=(Y_{i1},\dots,Y_{im})^{{}^{\top}} be a collection of m response variables from individual i . Let the corresponding lowercase letters \theta_{i},y_{ij}, and \mathbf{y}_{i} indicate the realizations of \Theta_{i},Y_{ij}, and \mathbf{Y}_{i} , respectively. Conditioned on \Theta_{i}=\theta_{i} , it is assumed that Y_{ij} , j=1,\dots,m , are independent (i.e., local independence; McDonald (1981a) ).

Read the complete text