ICCDesign: An R Package for the Design and Analysis of ICC-Based Reliability Studies with Continuous Responses

Ziyu Liu, Yundan Zhang, Ruilin Ma, Chenge Gao · 2026

Read ICCDesign: An R Package for the Design and Analysis of ICC-Based Reliability Studies with Continuous Responses by Ziyu Liu, Yundan Zhang, Ruilin Ma, Chenge Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02059.txt.

Introduction

Liu and co-authors package tools for studies that use the ICC statistic. The intraclass correlation coefficient is a staple of reliability work. It is central in medical measurement and psychology. Behavioral science also leans on it a lot. Two obstacles complicate its practical use today.

ICC comes in many forms under one classic framework. Six standard forms and four design mixes exist. The right form depends on the study design at hand. Guidance is not always built into software menus. Tools for sample size, estimation, and evaluation are fragmented. Users must switch between separate R packages. Their ICCDesign package unites design and analysis in one place. It aims to cut errors from that tool switching.

Catalog record

CREATORZiyu Liu, Yundan Zhang, Ruilin Ma, Chenge Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02059

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ICCDesign: An R Package for the Design and Analysis of ICC-Based Reliability Studies with Continuous Responses Ziyu Liu, Yundan Zhang, Ruilin Ma, Chenge Gao Abstract The intraclass correlation coefficient (ICC) is among the most widely used statistics in reliability research, playing a central role in medical measurement, psychological assessment, and behavioral science. However, practical application of ICC faces two major obstacles. First, ICC can be organized into multiple forms under the McGraw and Wong (1996) framework—including six widely reported standard forms and four additional design combinations—and researchers must select the appropriate form based on their study design, yet existing guidelines are not always operationalized in software interfaces. Second, available R tools are highly fragmented: sample size calculation, ICC estimation with confidence intervals, and reliability evaluation are distributed across separate packages, compelling researchers to switch between tools and increasing the risk of analytical errors. This paper introduces the ICCDesign package, designed specifically to provide an integrated workflow for ICC-based reliability studies with continuous responses, assuming one continuous rating per subject–rater cell. The package integrates four core functionalities: (1) point estimation, ANOVA-based confidence intervals, and implemented hypothesis tests for the ten supported ICC design settings following the McGraw and Wong (1996) framework, with a built-in four-step decision framework guiding users toward an appropriate ICC form; (2) sample size planning based on Zou’s (2012) closed-form formulas, supporting two planning modes and an inverse assurance calculation; (3) automated reliability evaluation based on Koo and Li (2016) criteria, with an uncertainty notification when the confidence interval spans the 0.75 good-reliability threshold; and (4) an interactive Shiny web application covering the main analysis and planning functionalities. ICCDesign is available from GitHub at https://github.com/KlariZhang/ICCDesign. 1. Introduction In repeated measurement studies, inter-rater agreement studies, and measurement instrument validation, researchers require a statistic that can quantify reliability among repeated measurements on the same object. The intraclass correlation coefficient (ICC) serves this purpose: built upon a variance components model, it partitions observed variability into subject-related and error-related components—with additional rater-related components in two-way designs. Depending on the selected form, ICC can quantify either absolute agreement or relative consistency among raters. ICC has been widely applied to evaluate measurement agreement in medical imaging, assess inter-rater reliability of clinical scales, and analyze test-retest reliability of psychological instruments (Shrout & Fleiss, 1979; McGraw & Wong, 1996; Koo & Li, 2016). Despite its widespread use, the correct application of ICC presents substantial challenges in practice. Shrout and Fleiss (1979) first systematically categorized ICC into three model forms, and McGraw and Wong (1996) subsequently organized these into ten design combinations, each corresponding to different research design assumptions. Selecting the appropriate ICC form requires researchers to address four core questions: whether the same set of raters evaluates all subjects; whether rater effects are treated as random or fixed; whether the unit of analysis is a single rating or the mean of multiple ratings; and whether the target is absolute agreement or relative consistency. Different answers correspond to different statistical models and computation formulas, and incorrect selection may lead to misleading reliability interpretations. It should be noted that ICCDesign does not propose a new ICC estimator; it consolidates established ICC estimation, inference, planning, and interpretation procedures into a unified interface. Within the R software ecosystem, existing ICC-related tools are distributed in a fragmented manner. The irr package (Gamer et al., 2019) and the psych package (Revelle, 2024) provide point estimates and confidence intervals for ICC, but cover only a subset of ICC forms and offer no guidance for form selection. The ICC.Sample.Size package (Wolak et al., 2012), based on the method of Zou (2012), provides sample size calculation but supports only a single calculation mode and contains no ICC estimation functionality. The irrNA package (Brueckl, 2022) extends irr to handle randomly incomplete datasets but likewise lacks design-stage functionality. This fragmentation means that conducting a complete ICC reliability study requires researchers to coordinate at least two or three separate packages, while inconsistent interfaces across packages increase the risk of operational errors. The ICCDesign package introduced in this paper aims to fill these gaps by providing an integrated solution for ICC-based reliability studies with continuous responses. The main contributions of the package are as follows: • Implementation of point estimation, ANOVA-based confidence intervals, and hypothesis testing for the ten supported ICC design combinations under the McGraw and Wong (1996) framework, covering six standard forms and four additional combinations; • A built-in four-step decision framework that guides users from study design specifications toward an appropriate ICC form through structured parameter settings, with explicit user feedback for special cases (automatic mapping and use-with-caution combinations); • Sample size planning based on Zou’s (2012) closed-form formulas, together with inverse assurance calculations; • Automated reliability grading integrated with the Koo and Li (2016) criteria, with automatic generation of standardized text-format analysis reports; • An interactive Shiny web application reducing the need to write R code once the package is installed. The remainder of this paper is organized as follows. Section 2 reviews the statistical foundations of ICC and describes the methods implemented in ICCDesign. Section 3 describes the package structure and core functionalities in detail. Section 4 presents a systematic comparison of ICCDesign with existing R packages. Section 5 demonstrates the use of the package through concrete examples. Section 6 provides discussion and directions for future development. 2. Methodological Background 2.1 ANOVA Framework and Three Statistical Models The statistical definition of ICC is built upon the analysis of variance (ANOVA) framework. Let n denote the number of subjects and k the number of raters, with Y_{ij} representing the continuous rating assigned to subject i by rater j . The current implementation assumes one rating per subject–rater cell and a balanced data matrix after preprocessing. ICCDesign implements three statistical models corresponding to different research design settings. The following mean squares are used throughout: MSR (subject mean square), MSC (rater/column mean square), MSE (residual/error mean square), and MSW (within-group mean square for the one-way model). One-way random effects model. This model is used when raters are not treated as a fixed common set across all subjects, corresponding to ICC(1,x) types. This corresponds to designs in which rater effects cannot be separated as a crossed factor: Y_{ij}=\mu+\alpha_{i}+\varepsilon_{ij} where \mu is the overall mean, \alpha_{i}\sim\mathcal{N}(0,\sigma_{\alpha}^{2}) is the subject random effect, and \varepsilon_{ij}\sim\mathcal{N}(0,\sigma_{\varepsilon}^{2}) is the residual error. The expected mean squares are: E(\text{MSR})=\sigma_{\varepsilon}^{2}+k\sigma_{\alpha}^{2},\quad E(\text{MSW})=\sigma_{\varepsilon}^{2} Two-way random effects model. This model is used when all subjects are rated by the same set of raters who are treated as randomly sampled from a broader rater population, corresponding to ICC(2,x) types. Since each subject–rater cell contains only one observation, the subject-by-rater interaction and pure measurement error cannot be separated. The combined residual term e_{ij} absorbs both sources: Y_{ij}=\mu+\alpha_{i}+\beta_{j}+e_{ij} where \beta_{j}\sim\mathcal{N}(0,\sigma_{\beta}^{2}) is the rater random effect and e_{ij} represents the combined interaction and measurement error. The expected mean squares are: E(\text{MSR})=\sigma_{e}^{2}+k\sigma_{\alpha}^{2},\quad E(\text{MSC})=\sigma_{e}^{2}+n\sigma_{\beta}^{2},\quad E(\text{MSE})=\sigma_{e}^{2} where \sigma_{e}^{2} denotes the combined residual variance. Note that E(\text{MSC}) applies to the two-way random effects model only; in the two-way mixed effects model, rater effects are fixed and \sigma_{\beta}^{2} is not defined. Two-way mixed effects model. This model is used when the raters constitute a specific fixed group, corresponding to ICC(3,x) types: Y_{ij}=\mu+\alpha_{i}+\beta_{j}+e_{ij} where \beta_{j} are fixed effects satisfying

Read the complete text