How much habituation is enough? An R application for ad hoc content-related limits of acceptance in behavioral research Konstantin Warneke, Stanislav D Siegel, José Afonso, Marco Herbsleb, Sebastian Wallot Abstract Evaluating test–retest reliability is of crucial importance for clinical decision-making and an integral component of studies in empirical sciences. In psychology, classical test theory focused on reliability coefficients, but research has shown that intraclass correlation coefficients and Cronbach’s alpha have limited value in quantifying individual measurement error sources appropriately. Preliminary assumptions such as parallel readings and independency of error sources are violated by practice/habituation effects, i.e., test–retest systematic errors. Although awareness of the need for additional measurement quantifications to properly assess test–retest agreement has increased, many such quantifications are insufficiently implemented. Furthermore, one of the most important agreement analyses introduced by Bland and Altman in 1986 has interpretation limitations, as the limits of agreement are descriptive and not linked to content-based clinical decision-making, oftentimes causing inappropriate usage. To counteract these limitations and improve the implementation of agreement analyses in addition to more commonly used reliability coefficients, we introduce a software package that allows predefining acceptable measurement limits (limits of acceptance) to evaluate the sufficiency of task habituation to gauge individual error sources. The package implements a comprehensive test–retest evaluation in order to improve transparency of data quality reporting in psychological and cognitive research. Introduction In empirical sciences such as psychology or neurocognitive research, reliability is a necessary, but insufficient, precondition for establishing internal and external validity of results (Patino & Ferreira, ). Broadly, reliability refers to the consistency of a measurement tool, procedure, or device in capturing the same characteristic under (mostly) identical conditions (Barnhart et al., ; Bartlett & Frost, ). Lachin () defines it as the extent to which a measurement is reproducible and free from errors. Originally, reliability was defined as the ratio of the true score variance to the observed total score variance (Cronbach et al., ; Lord & Novick, ). It was developed for assessing the within-observer error in repeated measures under identical conditions (i.e., test–retest reliability), or to identify the discrepancy between two readings from two different instruments (i.e., validity, assessing how an instrument compares to a gold standard) or between observers on the same subject (i.e., objectivity or inter-rater reliability) (Barnhart et al., ). Appropriate quantification of reliability is decisive in empirical research, where measurement error is inherent. In psychological and cognitive testing, a prominent approach for reliability evaluation adopts classical test theory (CTT), assuming X=T+E, with X being the observed score, resulting from the sum of a true score T and random error E (Medvedeva et al., ; Miller, ). Herein, the reliability coefficient is the ratio of the “true score” variance (σT2) to the “observed score” variance (σX2). Four assumptions should be fulfilled: 1) T and E are additive; 2) T and E are independent; 3) E is random; and 4) in repeated measures, T does not change (Collins, ). Krabbe () described that readings must be parallel, meaning that each assessed item (e.g., question, statement) was valid to evaluate the intended construct and no significant test–retest error occurred. Statistically, this means that there is no significant difference between the test–retest data and that the only occurring measurement error is random (Barnhart et al., ; Bartlett & Frost, ; Krabbe, ). By definition, the calculation of the reliability coefficient assumed the presence of a stable “true” score value (Barnhart et al., ; Collins, , p. 9). In practice, these assumptions are oftentimes contradicted by practice/test habituation effects, causing a significant and systematic test–retest performance improvement (Del Rossi et al., ; Donath et al., ; Lo et al., ). Additionally, the assumption of the error variance being an independent and single factor (Medvedeva et al., ) was questioned by the Generalizability theory (GT), an extension of the CTT that splits the error variance into several sources and accounts for multiple origins (Barnhart et al., ; Cronbach et al., ; Medvedeva et al., ). Common correlation-based reliability coefficients like Cronbach’s α (Collins, ) or intraclass correlation coefficients (ICC) (Bogdanov et al., ; Koo & Li, ) limit interpretability of the complete construct of test–retest reliability, as the pure reviewing of these coefficients prohibits a detailed perspective of different relevant measurement error sources (Kottner & Streiner, ; Warneke et al., ). These limitations can be counteracted by the implementation of measurement error analyses as a supplement to reliability coefficients, accounting for different error sources that are commonly conflated in typical test–retest reliability assessments (Atkinson & Nevill, ; Bogdanov et al., ; Hopkins, ; Warneke et al., ). In the following, we will briefly summarize different sources of measurement errors occurring in cognitive and behavioral research to underline the relevance of test–retest reliability with respect to internal data validity. Afterward, we will dive into the statistical quantification approaches and propose solutions to some of their limitations. Definitions and terms in test–retest reliability research Since reliability is a general term and construct, oftentimes used in numerous settings to describe specific constructs, we will briefly revisit definitions of terms and constructs that will be used in this work, while calculation formulas will be provided later in this work to avoid redundancies. Reliability is originally defined as an estimate of the true measurement construct that accounts for variability and error across multiple independent readings (Brennan, ; Cronbach, ). In its most intuitive sense, reliability simply means that a new measurement or retest value is the same as the truth or agrees with the truth (Barnhart et al., ). This is in accordance with the abovementioned test–retest variance splitting in the true score variance and the random error variance in CTT. However, a true score is not always measurable in test–retest evaluations. In contrast, in test–retest reliability analyses, as well as in interrater reliability assessment, the true score is hardly accessible, and oftentimes readings are not parallel, as systematic test–retest discrepancies are present. In this work, we will focus on test–retest reliability, although numerous aspects might fit other reliability constructs. A term oftentimes used as a synonym of reliability is repeatability, which seeks to quantify the error occurring when repeating the measurement directly in a row, without changing any circumstances or “the variation in repeated measurements made on the same subject under identical conditions” (Bartlett & Frost, , p. 467). When performed directly in a row, it is considered the purest form of assessing the closeness of agreement between the test and the retest (Barnhart et al., ; Food and Drug Administration (FDA), ; International Organization of Standardization, ). This describes a specific case of measurement precision. More broadly, precision could be defined as the closeness of agreement under stipulated or identical conditions (Barnhart et al., ). However, precision can be large despite readings deviating from the true value. This differs from accuracy, which refers to precisely hitting the right spot and depends on the existence of a gold standard against which the test can be compared. Both constructs are graphically illustrated in Fig. using target shooting to underline the difference. Because no generally accepted definition of reliability exists, Table summarizes how the term is used and the corresponding calculation formulas synthesized from the included literature. Statistically, precision mainly relates to the magnitude of random error, oftentimes stemming from standardization limitations and/or biological variability (Atkinson & Nevill, ; Warneke et al., ). When reaching the level of significance or reflecting a clinically relevant discrepancy, the systematic error compromises the assumption of parallel readings (Krabbe, ; Miller, ) and contradicts the calculation of reliability coefficients that assume no systematic discrepancy (Bartlett & Frost, ). A typical root cause of systematic errors in psychological or cognitive sciences is practice effects that affect test–retest reliability (Donath et al., ). Practice effects and systematic bias Test–retest improvements in coordinative, cognitive, or psychological readings without performing any intervention between the test occasions are known as practice- or habituation effects (Schworer et al., ; Warneke et al., ; White et al., ), usually occurring when participants are confronted with unfamiliar testing conditions (Collie et al., ; Estevis et al., ). Habituation effects (i.e., test–retest improvements within days or weeks in the absence of any intervention) are usually more pronounced between the first test–retest comparisons. Short-term, uninstructed test-to-test performance improvements over days or even weeks have been reported for a wide range of cognitive and executive function measures, including visuomotor tasks, interference inhibition tasks, and selective attention tests such as the Simon task, Eriksen flanker task, Stroop test, Wechsler Memory Scale, Digit Symbol Substitution Test, and Rapid Screen of Concussion. These studies show the largest gains between the first testing sessions, followed by a progressively flatter curve as participants become more familiar with the tasks, until no significant mean bias is observed between later testing days (Eileen De Monte et al., ; Theisen et al., ; Warneke et al., ; White et al., ). Similarly, long-term task learning effects from previous task performances up to years ago affected test scores in tests that were performed partly years after (e.g. the Halstead-Reitan Neuropsychological Test battery (Dikmen et al., ), the Wechsler Adult Intelligence Scale Fourth Edition (WAIS-IV) (Estevis et al., ) or the Wechsler Memory Scale III (Lo et al., ), showing the relevance of assessing specific task experience when testing for cognitive function (Collie et al., ; Estevis et al., ). These findings underline the relevance of sufficient familiarization with unhabituated tasks to avoid confusion between habituation effects and genuine changes, which might meaningfully bias clinical derivations (Bennett et al., ; Hopkins, ; Theisen et al., ). Random errors and biological variance As acknowledged in CTT and GT, random errors play a significant role when quantifying reliability (e.g., test–retest reliability). Quantifying and minimizing measurement error is relevant to internal (data) consistency and validity. Random errors play an important role as they might stem from various sources that can be either controlled via test standardization (e.g., tests performed always at the same time of the day, with the same investigator, homogeneous sample with comparable task experience, etc.) or remain inherent (e.g., biological variability, changes in day-to-day situational influences) (Hopkins, ). Although measurement error cannot be avoided entirely, its appropriate quantification remains inconsistent. The quantification of the random error deserves enhanced attention, excellent reliability (e.g., ICC ≥ 0.9) (Koo & Li, ) can be accompanied by meaningful measurement errors, with a test–retest mean and maximum absolute test–retest error of 20% and 90%, respectively (Atkinson & Nevill, ; Lamb, ; Warneke et al., ). Accordingly, a detailed measurement error analysis (how close is the agreement between two measurements within a subject on the same scale) to improve interpretability and to better account for the different sources of measurement errors should supplement reliability coefficient reporting (Bogdanov et al., ; Galpin et al., ; Lempke et al., ; Lynall et al., ; Sanders et al., ; Wilke et al., ; Wilke & Vogel, ). In the following, the most common quantification approaches are synthesized from the literature. Test–retest reliability quantifications: Calculation of common metrics Before providing formulas for reliability metrics, we present a definition of some of the basic terms. Let Xit denote the observed score of the individual i at measurement time t, with i=1,⋯,n and t=1,⋯,k. In the present application, the focus is restricted to two repeated measurements; therefore, k=2, where Xi1 represents the first measurement and Xi2 represents the second measurement for individual i. For each measurement occasion, the arithmetic mean is calculated as Accordingly, the means of the first and second measurement times are obtained by setting t=1 and t=2, respectively. The corresponding standard deviation for each measurement occasion is Because the following discussion focuses on test–retest quantifications with two repeated measurements, a pooled standard deviation is calculated as This pooled standard deviation has the advantage that it does not depend on which of the two repeated measurements is considered the test and which one was the retest. For the quantification of systematic bias and agreement, individual paired differences and participant-wise means are calculated. The paired difference for participant i is whereas the participant-wise mean is Calculation of intraclass correlation coefficients (ICC) The underlying mathematics of the ICC reflects an analysis of variance (ANOVA) approach, which is expressed as a correlation coefficient. For the calculation of the ICC, the sum of squares of all included values (sum of squares total) is divided into between- and within-subject sum of squares (i.e., the repeated measurements). The exact calculation depends on the type of ICC, and the formulas will be presented below. There are six ICC variants commonly used in the literature: ICC(1,1), ICC(1,2), ICC(2,1), ICC(2,2), ICC(3,1), and ICC(3,2), that were presented by Shrout & Fleiss (Shrout & Fleiss, ) and subsequently summarized by Koo & Li () to the currently most-cited guidelines on how to choose the correct ICC. The choice for the correct ICC variant depends on three questions (Shrout & Fleiss, ): i) Is a one- or a two-way ANOVA preferable? ii) is there an important difference between judges’ mean ratings? iii) is the rating unit an individual rating or a mean of multiple ratings? In the present application, k=2; the notation k (i.e., number of repeated measurements per observation unit) is retained to make the distinction between single-measure and average-measure ICCs explicit. To derive the ICCs, participant means, measurement occasion means, and the grand mean are calculated first. The measurement occasion mean is defined in Eq. . The participant-wise mean is calculated as For the present case with two repeated measurements, this corresponds to Eq. . The grand mean across all observations is For ICC(1,1) and ICC(1,k), the application uses the one-way random-effects formulation. The between-individual sum of squares is the within-individual for the different measurement time points (here: k=2) sum of squares is and the total sum of squares is The residual error sum of squares is then The corresponding mean squares are and From these terms, the one-way ICCs are calculated as and These coefficients represent the reliability of a single measurement and the average of both measurements, respectively. Note that in the original publication from Shrout & Fleiss (), the SSindividual was SSbetween and SStime was SSwithin (the same applies to MS). To adhere to the index i (individual) and t (time) introduced above, the nomenclature was adopted for congruency. For the two-way ICCs, the application separates participant variance, measurement variance, and residual error variance. Using these quantities, the two-way random-effects absolute-agreement ICCs are calculated as and The corresponding two-way mixed-effects consistency ICCs are and These calculations correspond to the standard ANOVA-based ICC definitions and are consistent with the formulations used in established reliability literature (Koo & Li, ) and software implementations. To improve interpretability, the application also provides 95% confidence intervals for all ICCs. For ICC(1,1), ICC(1,k), ICC(2,1) and ICC(2,k), ICC(3,1), and ICC(3,k), the intervals are derived from the relevant F-ratio, transformed into ICC bounds, and are calculated as in the “irr” package (Gamer & Lemon, ). Cronbach’s alpha (α) A further, related quantification oftentimes used for internal consistency across the two repeated measurements is Cronbach’s α (Cronbach, ). For two repeated measurements, α was computed from the item-variance formulation where k=2, σj2 denotes the variance of measurement j, and σtotal2 denotes the variance of the summed score across both measurements (Bland & Altman, ). Calculation of the standard error of measurement (SEoM) and minimal detectable change (MDC) The SEoM (also oftentimes referred to as SEM (Harvill, ; Weir, )) quantifies the expected magnitude of measurement error in the original unit of measurement, including the ICC in its calculation. In the presented work, the nomenclature is slightly adjusted to SEoM, as the abbreviation of SEM is also used for structural equation modeling). In the present application, the SEoM (Harvill, ; Warneke et al., ) is calculated as Based on the SEoM, the MDC (Seamon et al., ) at the 95% confidence level is then calculated as The SEoM reflects the typical random measurement error, whereas the MDC indicates the minimum change required to exceed measurement error with 95% confidence. This logic is closely linked to the interpretation of real change in repeated-measures designs. As outlined above, excellent reliability (e.g., ICC ≥ 0.9) (Koo & Li, ) can be accompanied by meaningful measurement errors with a test–retest mean and maximum absolute test–retest error of 20% and 90%, respectively (Atkinson & Nevill, ; Lamb, ; Warneke et al., ). Accordingly, several authors requested the supplementation of common reliability coefficients with an agreement analysis (i.e., how close is the agreement between two measurements within a subject on the same scale) to improve interpretability and to better account for the different sources of measurement errors (Atkinson & Nevill, ; Barnhart et al., ; Warneke et al., ). Agreement analysis in reliability research The BA analysis evaluates the closeness of agreement between two measurements under identical conditions within a subject (Doğan, ; Giavarina, ; Haghayegh et al., ) rather than providing a correlation-based analysis as a stand-alone quantification (Bland & Altman, , ; Bland & Altman, ; Doğan, ; Lin, ; Vetter & Schober, ). Therefore, BA analyses are an important complement to typical reliability analyses. While initially presented to compare a gold standard method against a novel measurement device (validity studies), it can also be used in a comparative way without having a gold standard to explore the test–retest error distribution (Bland & Altman, ). Agreement is evaluated by plotting the test–retest difference (y-axis) against the respective mean of the two measurements (x-axis). It graphically illustrates random scattering (random error) around the test–retest mean bias (systematic error) (Bland & Altman, ; Bogdanov et al., ; Warneke et al., ). If the mean of all test trials is larger than the mean of the retest trials, the mean bias line diverges from the zero-line and indicates a systematic error. The significance of this deviation can be tested via a paired-samples t test (see systematic error). If the summed random scattering approaches 0, no systematic error occurred. Note, however, that the result of this test – as with other assumption tests – is sample-size dependent. Hence, the significance of this test should not be blindly used as a sole criterion to evaluate bias without visual inspection. The random error is commonly quantified by plotting the 95% limits of agreement (LoA) reflecting the meandifferences±1.96×Standarddeviation, which provides a reference interval for test–retest differences expected for 95% of individuals of the analyzed population (Wright & Royston, ). According to Barnhart et al. (), the agreement analysis is therefore able to account for precision and relative accuracy. If individual data points are widely scattered without a systematic mean bias, test–retest reliability is hampered by imprecision. If the systematic error diverges significantly from the zero-line, an additional systematic error suggests an accuracy problem. An accuracy problem is indicated if the retest systematically lies above or below the test, which could be attributed to habituation effects, even if precision was sufficient. In that case, the retest precisely missed hitting the correct spot. Bland–Altman plots corresponding to target shooting (see Fig. ) are provided in Fig. . Underlying simulated data are provided in the Supplemental Material (Tables and ). Before calculating the ICC, Bartlett & Frost () suggested performing a BA analysis as a precondition for the validity of reliability coefficients that assume parallel readings. Here, it is noteworthy that large ICCs do not automatically refer to high accuracy (Fig. A) or precision (Fig. B), underscoring the call for supplementing ICCs via these analyses (Atkinson & Nevill, ). Calculation of Bland–Altman metrics (systematic bias, limits of agreement) As defined above, di (4) reflects the individual test–retest deviation, whereas the value mi (5) serves as the horizontal coordinate in the BA plot. The systematic bias is assessed by the mean of the paired differences, where a paired-samples t test can examine if the mean difference differs significantly from zero. The standard deviation of the paired differences is The corresponding t-statistic is A significant result indicates that one measurement tends to be systematically higher or lower than the other. This distinction is important because large relative reliability can still coexist with meaningful systematic bias (Warneke et al., ). The LoAs are calculated as and How to appropriately interpret Bland–Altman plots Given the nature of BA analyses, LoAs should be assessed relative to the respective mean value. The larger the y-axis range in a BA plot, the lower the precision of the measurement. Reporting LoAs without referring to the mean of the measurement values can lead to misleading conclusions (Grgic et al., ). For instance, visual vertical perception is clinically relevant in subacute stroke patients. In one study (Piscicelli et al., ), BA analyses were performed between the test–retest values. The authors reported LoAs were between – 1.03 and 1.33°. Without providing a reference mean for the random error, the result is difficult to interpret. For instance, if the mean was 0.5°, this would refer to a large measurement error surpassing 100%. However, if the mean was 180°, the measurement error would be less than 1%. Although lower and upper LoAs allow for quantification, several articles miscommunicate BA analyses and LoAs and state that sufficient reliability was reached as most data points were within the LoAs (for instance: (Brandl et al., ; Palsma et al., ), or provide LoA quantification without a mean reference, which does not allow for adequate interpretability. Since BA analysis plots the mean differences between two values, LoAs must be interpreted in relation to the reference mean of the test–retest. A more appropriate reporting practice is expressing the upper and lower LoAs in relation to the respective mean (Cataldi et al., ). Another possibility to receive information about the measurement error is the reporting of the mean absolute error (MAE) (Willmott & Matsuura, ), the mean absolute percentage error or the adjusted MAPE (MAPEadj) (Warneke et al., ). An alternative is to report the adjusted mean absolute percentage error (MAPE) (Warneke et al., ), which is calculated as follows: The MAE reflects the average absolute test–retest deviation in the original unit of measurement and is a straightforward and interpretable summary of average error magnitude. It is defined as Since the MAE must be reported in combination with the respective measurement mean, the mean absolute percentage error (MAPE) can be calculated as a dimensionless quantification Since the MAPE in its original form depends on the reference mean (here Xi1) (as above: i stands for individual and t stands for time, and in this formula, t=1), the adjusted MAPE can be used to provide a potential solution, which is calculated based on the measurement error calculations in BA analyses. Warneke et al. () proposed: Accordingly, MAPE expresses the average absolute test–retest deviation relative to the subject-wise mean and reports it as a percentage. This can facilitate comparisons across variables with different units or scales. Practice effects, reliability, and agreement analyses In the previous sections, we outlined that practice/habituation effects result in a significant test–retest improvement in numerous cognitive tasks, such as the Eriksen flanker test, the Stroop test, Digit Span Test, or the Trail Making Test, without performing an intervention between trials. Donath et al. () showed that habituation improves reliability coefficients in executive function tests, and our group demonstrated that test practicing reduced the variability of cognitive performance over time, and systematic task habituation effects can be accompanied by a trial-to-trial reduction in the random error (reference included after acceptance to keep blinding). Therefore, measurement errors are not automatically independent as assumed in the CTT second assumption, and the random error is influenced by biological variability as well as practice effects. Warneke et al. (, ) repeated commonly performed neurocognitive tests twice a day for five consecutive days. Beyond the systematic bias, testing neurocognitive function (Trail Making Test, Stroop test, Choice reaction task) over five consecutive days twice per session started with trial to trial random errors of up to 6–43%, but stabilized below 5% in reaction tasks, 5–9% in interference inhibition and about 10% in visual scanning (Warneke et al., , ). Standard habituation procedures with only one practicing appointment before data collection started (if any exist) or performing some test trials before the main testing session started are often insufficient to produce reliable and valid testing results (White et al., ). Accordingly, more rigorous testing protocols in neurocognitive tests that take sex- (Reimers & Maylor, ; Warneke et al., ), age (Ramos-Loyo et al., ; Warneke et al., ) and task specific habituation courses with a critical dependence on previous task exposure (Collie et al., ; Estevis et al., ) into account are needed. Although there are several articles that assumed their tests were reliable, per se, the question of how many habituation sessions are sufficient to reach satisfying reliability is likely context- and population-specific (Warneke et al., ). As early as 1992, Thompson () criticized the common practice of assuming that a test is reliable per se as "sloppy speaking". The previously outlined results underscore that sufficient task habituation to improve test–retest reliability, repeatability (closeness of agreement), and accuracy of the test critically depends on the evaluated population and should specifically be determined for a given person or sample. As part of this paper, we developed an R-based software tool that will help with the evaluation of habituation effects and whether enough practice trials were implemented before the actual testing started. Advanced statistical approaches to improve transparency and interpretability of reliability metrics Atkinson & Nevill () noted more than 25 years ago: “While statistics will never be more important than a well-designed reliability study itself, it is sensible that there should be a standardized statistical analysis for any reliability study involving ratio of interval measurements” (p. 235). For reliability studies, but also for cross-sectional and longitudinal studies, reporting repeatability and measurement errors within the construct of test–retest reliability is a cornerstone in high-quality results reporting, as pointed out in the APA reporting standards for quantitative research (Appelbaum et al., ). As a viable supplement, and to evaluate whether the parallel readings assumption used to justify ICC calculations is met (Bartlett & Frost, ), cognitive and psychological research should add agreement analyses to ensure the precision and accuracy of data collection and transparent results reporting to the reader, in addition to common reliability coefficient-based reliability metrics. However, in its current form, BA analyses are descriptive, without content-related classification (Bland & Altmann, ; Giavarina, ) and do not allow for statistical inference statements about whether sufficient repeatability was achieved and if the measurement errors are within an acceptable range. BA analyses simply provide a post hoc description of measurement errors. To classify reliability as satisfying, acceptable limits must be defined a priori and consider clinical and practical requirements on accuracy (Giavarina, ; Warneke et al., ). In this work, we provide a tool to pre-set acceptable limits of acceptance (LoAcc) that differ from limits of agreement, implemented in a freely available R-based tool. This tool may help researchers to facilitate a comprehensive and standardized measurement error reporting across various research disciplines, including medicine, social sciences, and empirical natural sciences. Establishing limits of acceptance (LoAcc) to improve interpretation Bland and Altman () stated that “Ideally, it should be defined in advance to help in the interpretation of the method comparison and to choose the sample size.” (p.308). Accordingly, a content-based a priori limit determination is needed to evaluate whether sufficient reliability was reached in the data collection (Giavarina, ; Warneke et al., ). The limitation of using parallel LoAs for small and large mean values within one analysis is explored in Fig. . In many empirical research areas, the measurement scattering around the mean bias gets larger when the value of the measure increases (Hopkins, ; Nevill & Atkinson, ). However, LoAs are parallel. This misleadingly suggests a constant measurement error across the sample. In fact, in their current form, parallel lines in BA analyses for test–retest differences in dependency on their respective means tolerate higher percentage measurement errors for smaller means (left side of the BA plot) compared to larger means (left side of the BA plot). In other words, the parallel LoAs suggest accepting a larger percentage error for small test–retest mean values. This complicates the interpretation and ad hoc definition of content-based acceptable measurement ranges, as LoAs being parallel to the mean bias neglects percentage errors and assumes homoscedasticity (Bartlett & Frost, ). Both scenarios are illustrated in Fig. (A and B). A sole post hoc description of measurement errors with parallel LoAs stands in conflict with the idea of a priori and content-based precision and accuracy classifications to define which measurement errors (systematic and random errors) can be considered acceptable. Although individual studies have predefined acceptable LoAs (Taule et al., ), these were 1) only set to 2±SD of the mean difference without a content-related justification (automatically, 1.96×SD are satisfying) and 2) did not account for heteroscedasticity, a phenomenon in which most data points were within these predefined limits. In contrast, we suggest that expected percentage intervention-induced changes can serve as an orientation for which random measurement error can be considered appropriate to achieve sufficient precision. A systematic error is never acceptable for accurate measurements (and hampers the validity of the ICC (Bartlett & Frost, )). This may vary depending on the goals and scientific field, with some areas probably requiring narrower acceptable measurement error than others. This approach can substitute abstract reliability coefficients such as ICC or Cronbach’s α (Koo & Li, ) by providing content-based agreement analyses (Bland & Altman, ; Giavarina, ). Calculation of limits of acceptance (LoAcc) In contrast to the LoAs (see formulas –), the acceptable threshold is defined proportionally to the subject-wise mean: where p denotes the user-defined acceptable proportional error threshold and mi is the subject-wise mean of the two repeated measurements. Accordingly, a paired difference is classified as acceptable if The acceptable (absolute test–retest differences, see y-axis in the BA analysis) range increases with increasing measurement means, allowing the visualization and evaluation of proportional agreement limits in the presence of heteroscedasticity. Advancing reliability reporting To help with practical investigation of reliability, a Shiny app will be presented to automate the calculation of key reliability metrics in the following (Access: https://cloud.uni-jena.de/s/Zf8XW62zQeyssEy or via the Open Science Framework profile (osf.io/gbcea). These include Cronbach’s α (Bland & Altman, ), intra-class correlation (ICC) (Koo & Li, ), standard error of measurement (SEM) (Palta et al., ; Weir, ), and minimal detectable change (MDC) (Cesar et al., ; Seamon et al., ) for relative and absolute reliability. A descriptive way to quantify the actual random error is through the mean absolute error (MAE) and mean absolute percentage error (MAPE) (Warneke et al., ; Willmott & Matsuura, ), summing up the absolute test–retest differences by dividing them by the participant count in absolute terms and expressed as a percentage, respectively. Additionally, the script accounts for systematic bias by performing a paired-samples t test (Atkinson & Nevill, ). The Shiny app enhances the interpretability of reliability metrics, so that practitioners and clinicians can easily review and interpret study results in light of the measurement error. Therefore, the script allows users to define pre-set LoAcc, providing a structured approach for evaluating acceptable measurement errors in their dataset. Both features improve the clarity of measurement error assessment and facilitate more standardized reporting practices. In the following section, we will describe in detail an example of how the script works and what formulas are used. LoA versus LoAcc in neurocognitive tests: An example of reliability evaluation, including reliability coefficients as well as agreement analysis Assuming that practice or task habituation compromises the internal validity of a data set via meaningful systematic and random errors, the quantification of both error sources can be performed via BA analyses. Therefore, in the first step, an acceptable, content-based measurement error range must be defined (Bland & Altman, ; Giavarina, ; Warneke et al., ). The classification of measurement errors as being clinically or practically relevant depends on a variety of factors, but pre-post effects extracted from previous studies (with comparable samples and experimental settings) could be used to define an acceptable range. The appropriate pre-set measurement error A measurement error such as a coefficient of variation ≤10% (Cabarkapa et al., ; Luque Laguna et al., ) is often described as acceptable, but we propose further contextualization. For instance, if an intervention results in a pre-post change of 100%, a (random) measurement error of 10% might be acceptable. In contrast, if an intervention effect of 10% is assumed, a measurement error in the pre- and post-test of 10% is unacceptable. Therefore, authors should make this decision transparently in studies to communicate to the reader which measurement error was tolerated. After defining a tolerable measurement error for test–retest deviations, data can be promptly analyzed. The choice of the acceptable measurement error margin strongly depends on the study, research area and goal. For instance, Jogie et al. () allowed a maximum difference of ±10 in the Goal Attainment Scale and the Functional Independence Measure in the agreement analysis between different observers. In blood pressure measurements, Gibbon et al. () pre-set a limit of ±5 mmHg, and Hwang et al. () predefined ±2 units in the Rowland Universal Dementia Assessment Scale and the Geriatric Depression Scale. In a previous study, Warneke et al. () showed that task-to-task habituation diminished the systematic and random error until both stabilized at a significantly reduced level. Since no generally implemented guidelines on which level of measurement error is acceptable exist, this example relies on the magnitude of MAPE observed after four days of habituation as outlined in Warneke et al. (), with 4.66% MAPE. Therefore, the acceptable measurement error was set to 5% to illustrate how the Shiny app works. To illustrate, imagine a Stroop test was performed on four consecutive days as described in the latter study. Note that the user of the Shiny app can pre-set the acceptable measurement error margin to any percentage value that makes sense from a content-related perspective. Data structure Before running the application, the dataset must be arranged in a standardized wide format. Because the present workflow is restricted to pairwise reliability analysis, all implemented reliability indices are calculated from exactly two repeated measurements, for example, test versus retest (test–retest reliability). It can also be applied for objectivity (if one column contains values from rater 1 and the other rater 2) or validity (column 1 is the gold standard, column 2 the device that should be evaluated). To ensure error-free import and processing of the data, the first column must contain the participant identifier, whereas all subsequent columns must contain measured variables. These variables can be arranged pairwise in advance, because the script automatically treats two consecutive columns as one measurement pair. Thus, columns 2 and 3 form the first pair, columns 4 and 5 form the second pair, etc. Alternatively, measurement pairs can also be selected manually within the application (see Fig. ). In the present example, we simulated Stroop test data that can be accessed for transparency in the Supplementary Material Excel file Data_plot2). The first column contains the participant identifier (ID), whereas each test day is represented by two separate columns containing the repeated measurements (e.g., day 1 and day 2). Steps through the Shiny app The Shiny app can be started in RStudio by running the script. The script first installs and loads all required packages. Once the application window opens, the user can review the complete code used for maximal transparency and, once the data set was uploaded (see first step in the description), the user is guided through the analysis procedure step by step. Therefore, for inline code, we refer to the uploaded file, usable via the following link: https://cloud.uni-jena.de/s/Zf8XW62zQeyssEy or made accessible via our Open Science Framework profile (osf.io/gbcea). The following tutorial will guide the potential user through the application process: Run the application: After copying the Code into RStudio, the user can review the individual steps of the application with some commentary and instructions on the RStudio surface. By clicking on “Run App” the analysis is initiated (see Fig. ) Upload the Excel file: containing the data and select the relevant worksheet. To facilitate verification of the data structure, the application provides a preview of the imported file (i.e., names of the first rows and columns, file name, sheet name, dataset dimensions, and the number of missing values per column) (see Fig. ). Define the measurement pairs: If the variables were arranged pairwise in advance, the application can assign the corresponding column pairs automatically. The selected columns can also be modified manually. At this stage, the user additionally specifies the LoAcc for each variable pair. A default value of 0.05, corresponding to an acceptable random scattering/random error of 5% in our example. This value can be changed according to the user’s preference. The user may also optionally define the measurement unit for each variable pair (e.g., ms, N, °), which will then be displayed in square brackets in the x- and y-axis labels of the Bland–Altman plots (see Fig. ). Select output metrics: Based on Atkinson & Nevill (), our package can calculate the most common metrics used in test–retest reliability and agreement analyses such as Cronbach’s α, ICC, SEM, MDC, CV, MAE, MAPE, limits of agreement, systematic bias, among others. To account for different study and evaluation settings, all available ICCs can be evaluated with the SEM and MDC using the respective ICC version (see Fig. ). Available quantifications and how they were calculated are described in the following section “Calculation of metrics” in detail. Define the export and plot settings: These settings include the plot mode (color or black and white), the image dimensions, and the output directories for the Excel results file and the plot files. In addition, the user can define whether the Bland–Altman plots should be saved as individual figures, as one combined multi-panel figure, or in both formats. If a combined Fig. is selected, the panel layout (rows × columns) can also be specified. Default settings are pre-programmed, including a colored plot with a resolution of 800 × 600 pixels and export paths corresponding to the current working directory (see Fig. ). Start the analysis: The application then displays the selected test–retest reliability metrics in tabular form and provides the corresponding BA plots within the graphical interface (see Fig. ). In addition, the results are exported automatically. If desired, the user may subsequently return to earlier steps, modify the settings, analyze another file or worksheet, or close the application. Results interpretation of the data simulation outcome To make the different scenarios visible, we simulated a data set that could illustrate Stroop color word test accomplishment times. The data were generated to illustrate an optimized task habituation quantification process to illustrate the examples. Therefore, four testing days were generated, and the test–retest analysis, including the described metrics, was performed from the initial day (day1) to the second, the second to the third, and the third to the fourth day. As previously described, the LoAcc are not displayed as parallel horizontal lines. Instead, they are adjusted to the magnitude of the respective test–retest mean, because fixed absolute limits would permit disproportionately large percentage errors at small means, which appears unreasonable. Thus, acceptable percentage test–retest differences in the BA analysis require proportional limits that increase with increasing mean values. This limit was exemplary set to 5% in this analysis. In the comparison between day 1 and day 2 (Fig. A), the data indicate insufficient accuracy. Mean performance increased from 26.56 ± 0.98 to 27.37 ± 0.92, and the bias test was significant (p < 0.001), indicating systematic error. Consistently, the bias line is shown in red. In addition, 86 of 100 observations fell within the predefined LoAcc, although 95 of 100 would have been required to satisfy the 5% criterion. The LoAcc are therefore also displayed in red, indicating insufficient precision. The MAE was 0.828 s, and the 95% limits of agreement ranged from − 0.159 to 1.772 s. Together, these metrics indicate that both systematic error and random scatter were outside the acceptable range. For day 2 vs. day 3 (Fig. B), the results suggest partial habituation. Mean performance changed only slightly from 27.37 ± 0.92 to 27.52 ± 1.21, and the bias test was no longer statistically significant (p = 0.053), suggesting that the systematic shift had largely disappeared; accordingly, the bias line is shown in green. However, random error remained too high, as 88 of 100 observations were within the LoAcc, so the LoAcc remained red. Although the MAE decreased to 0.610 s, the predefined precision criterion was still not met. Thus, Fig. B reflects improved, yet still insufficient, measurement precision. Importantly, the output provided by the R package also allows a more differentiated interpretation of such cases, because a slight deviation from the acceptance threshold may be judged differently from a substantial failure to meet the criterion. In the comparison between day 3 and day 4 (Fig. C), the data fulfilled the predefined criterion for acceptable accuracy. Mean performance remained stable (27.52 ± 1.21 vs. 27.48 ± 0.96), the bias test was non-significant (p = 0.463), and 97 of 100 observations fell within the LoAcc. Only three observations were outside the acceptable range, which is below the tolerated number of five. The MAE decreased further to 0.479 s, and the 95% limits of agreement narrowed to − 1.239 to 1.149 s, indicating that both systematic and random error had decreased to an acceptable level. In this case, both the bias line and the LoAcc are shown in green. The ICCs showed the same general pattern, increasing across the consecutive comparisons and thereby indicating improved stability over sessions. Similar tendencies were also observed for SEM and MDC, which were lowest in the final comparison. Overall, Fig. illustrates a progression from insufficient accuracy with both systematic and random error, to partial habituation with reduced systematic error but still excessive random scatter, and finally to acceptable measurement quality once both error sources fall within the predefined limits. It should also be noted that with progressing habituation, the conventional LoA and the LoAcc become more closely aligned, whereas in the first comparison (Fig. A) they differ more markedly. This can be explained by two factors. First, systematic bias shifts the data vertically, and the LoA are always distributed symmetrically around the mean bias. Second, reduced precision increases the spread of the data and thus widens the LoA. In the final comparison, the systematic error had largely dissipated, and the random scatter had decreased, resulting in a closer correspondence between LoA and LoAcc. This again illustrates that the LoA are descriptive in nature, whereas the LoAcc provide an additional content-based classification criterion defined a priori (see Fig. ). The outcome table is provided in Table . To illustrate the applicability of the script for real empirical data, we applied the R script to one of our previously published datasets examining the impact of task habituation on systematic error, random error, and test–retest reliability coefficients in, among others, the choice reaction task across five testing days (Warneke et al., ; Ethics No. GZ.39/163/63). To demonstrate how the predefined LoAcc influences interpretation, we analyzed the data using both a 5% and a 10% threshold (although commonly provided as the coefficient of variability), as these thresholds provide a baseline for a potential quantification, which is discussed in the literature across different disciplines (Currell & Jeukendrup, ; Marenco et al., ; Westheim et al., ). Whether a 5% (Fig. ) or 10% (Fig. ) limit is applicable might remain disputable as being arbitrary (Atkinson & Nevill, ), but is not the focus of the present work. The resulting plots are shown in Figs. and , and the corresponding output for both 5% and 10% LoAccs is presented in Table . Notably, the 10% threshold (Fig. ) also illustrates a near-threshold case: three data points were allowed to fall outside the LoAcc, but this criterion was exceeded by one point lying exactly on the limit. Thus, although the LoAcc are still displayed in red, the 10% threshold was almost satisfied, demonstrating that such classifications should always be interpreted in light of the magnitude and context of the deviation rather than as a purely binary decision. Conclusion For a comprehensive measurement evaluation, the systematic test–retest error (e.g., via mean bias line in BA analyses supplemented via paired-sample t test) as well as a random error analysis (e.g., via the MAE or MAPEadj should supplement the reliability coefficient (e.g., ICC) quantification. Both, the systematic and random errors can be graphically illustrated via BA. Specifically, the difference between the 0-line and the actual average indicates the magnitude of systematic error , which is important to consider because a valid interpretation of ICCs requires the absence of systematic errors (Bartlett & Frost, ). Also, the MAE is confounded in the presence of a systematic error, as the random and systematic error messes up and might misleadingly indicate a large random error, as the MAE considers the absolute test–retest differences. The additional presentation of LoAcc allows for an ad hoc content-related evaluation of both error sources and therefore solves open limitations in common BA analyses in which the LoAs all-time summarized the mean bias line (indicating a dependency). Furthermore, it facilitates the calculation and interpretation of further relevant reliability quantifications. However, to reach appropriate accuracy, the systematic bias should be viewed as an independent error source (also: see CTT), and both the systematic and the random error should be minimal. Only if both requirements are fulfilled independently, sufficient accuracy is indicated. Measurement errors inherently occur in empirical data collections in a broad span of research topics. When seeking to ensure internal data validity, habituation effects must be controlled, and it is necessary to distinguish between systematic and random errors, which should be made transparent for reasonable data interpretation. The sole post-hoc and content-free agreement analysis of BA plots does not fulfill these requirements. By setting predefined acceptance criteria, researchers enhance the interpretability and applicability of BA analyses, making results more meaningful for scientific and practical use. To counteract this, this work introduced an easily applicable and generalized approach to define acceptable measurement error limits ad hoc. These allow for a detailed analysis of systematic and random errors in terms of test–retest reliability as a complement to commonly applied BA analyses. Although the main objective here is to improve test–retest reliability reporting in which an agreement analysis is rather comparative than absolute, pre-set LoAccs can also be applied in validity or objectivity quantification, when having a gold standard measurement as a reference, or when comparing readings from different raters. The opportunity was previously outlined by Bland and Altman () when introducing BA analyses and also apply to the introduced Shiny App. To allow for objectivity quantifications, the respective ICC(2,1 and 2,k) should be selected (Koo & Li, ; Shrout & Fleiss, ). Limitations Although the software package automates the test–retest evaluation and allows pre-definition of acceptable measurement errors, one main concern that remains unsolved is the missing content-based classification of what can be considered acceptable. However, such acceptable error classification critically depends on the respective objectives of the study, assumed pre-post changes or effect sizes, and, for sure, the measurement. Since it was neither the objective of this study, nor might it be possible to provide guidance on acceptable limits in this work, the used 5% and 10% pre-set LoAccs should be considered as an example. Nevertheless, using this software package allows for making transparent what authors assumed to be an acceptable test–retest measurement error as a complementation of already existing test–retest reliability quantifications, while future works should address how acceptable limits might be imputed from previous works, as it is done for, for instance, G-power sample size estimations, where previous effect sizes from related studies were used to calculated the required sample size. Another point of concern is that the application is limited to two columns. However, since this package was designed to improve the interpretability of BA analyses and allow for requested pre-set acceptable measurement error ranges, future work should address the query of implementing more than two data rows into an agreement analysis. Lastly, in the CTT, error sources (T and E) were considered to be independent from each other. Although considering this clear separation as a necessary precondition for reliability and repeatability quantification might be disputable (see Generalizability theory), this discussion about the most appropriate test theory was not extensively discussed in this study, while both sources of error were quantified via the Shiny app independently. Authors’ contributions KW – Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Validation, Writing – original draft, Writing – review & editing, SDS – Data curation, Methodology, Software, Validation, Writing – original draft, Writing – review & editing, performed the graphical illustrations, assisted in writing the first draft, JA – Formal analysis, Validation, Writing – review & editing, MH – Formal analysis, Validation, Writing – review & editing, SW – Formal analysis, Supervision, Validation, Writing – review & editing. All authors contributed significantly to the work and agreed to the final version. Footnotes