A categorization of performance measures for estimated non-linear associations between an outcome and continuous predictors
Theresa Ullmann, Georg Heinze, Michal Abrahamowicz, Aris Perperoglou, Willi Sauerbrei, Matthias Schmid, Daniela Dunkler, for TG2 of the STRATOS initiative · 2025
Read A categorization of performance measures for estimated non-linear associations between an outcome and continuous predictors by Theresa Ullmann, Georg Heinze, Michal Abrahamowicz, Aris Perperoglou, Willi Sauerbrei, Matthias Schmid, Daniela Dunkler, for TG2 of the STRATOS initiative (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16981.txt.
Introduction
Theresa Ullmann and co-authors sort the many ways to grade how well a fitted curve matches a true nonlinear link, in a paper from 2025. In health and social research, the link between a measure and an outcome is often curved, not straight. Flexible tools such as splines can fit such curves, and simulation studies compare those tools head to head.
Such comparisons need scores that grade how close a fitted curve sits to the true curve. Many scores have been proposed, but no orderly overview existed. The authors build a sorting scheme that holds the common scores and shows how they differ in what they reward. The work comes from a methods group focused on stronger analysis practice. Their scheme should help researchers pick scores with open eyes.
Catalog record
| CREATOR | Theresa Ullmann, Georg Heinze, Michal Abrahamowicz, Aris Perperoglou, Willi Sauerbrei, Matthias Schmid, Daniela Dunkler, for TG2 of the STRATOS initiative |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A categorization of performance measures for estimated non-linear associations between an outcome and continuous predictors Theresa Ullmann, Georg Heinze, Michal Abrahamowicz, Aris Perperoglou, Willi Sauerbrei, Matthias Schmid, Daniela Dunkler, for TG2 of the STRATOS initiative Abstract Abstract In regression analysis, associations between continuous predictors and the outcome are often assumed to be linear. However, modeling the associations as non-linear can improve model fit. Many flexible modeling techniques, like (fractional) polynomials and spline-based approaches, are available. Such methods can be systematically compared in simulation studies, which require suitable performance measures to evaluate the accuracy of the estimated curves against the true data-generating functions. Although various measures have been proposed in the literature, no systematic overview exists so far. To fill this gap, we introduce a categorization of performance measures for evaluating estimated non-linear associations between an outcome and continuous predictors. This categorization includes many commonly used measures. The measures can not only be used in simulation studies, but also in application studies to compare different estimates to each other. We further illustrate and compare the behavior of different performance measures through some examples and a Shiny app. Graphical Abstract Suppose there is a known true nonlinear association between X and Y (black curve), e.g., in a simulation study. We want to find the best possible approximation (colored curves) to that truth. However, there are many measures quantifying the similarity of each estimated curve to the ground truth. 1 Introduction Regression models often describe associations between an outcome and one or multiple predictors (also referred to as explanatory variables, covariates, or independent variables). Determining the functional forms of the continuous predictors in such models is an important aspect of model-building and may substantially affect the validity and interpretation of the model. Often, the associations of the continuous predictors with the outcome are simply assumed to be linear (potentially after transformation of the expected outcome via a link function). However, modeling the effect of one or multiple predictors as non-linear can be more appropriate in many applications. Consider the example in Fig 1 a) which shows the association of age with body mass index (BMI) based on data from 9377 participants of the National Health and Nutrition Examination Survey (NHANES) ( CDC “& NCHS, 2024a ). The black curve connects the mean BMI values at each age. To account for the sampling variation reflected in the excessive wiggliness of the black curve, we can try to find a smooth model fit to better describe the systematic changes in the mean BMI values with increasing age. The linear fit, shown in green color, does not describe the mean BMI values well (by definition, the linear fit is unable to account for the clear non-monotonicity of the relationship), such that options for a non-linear functional form should be explored. Figure 1: Association of age in years with BMI, estimated from 9377 participants of NHANES. The black line connects the mean BMI values at each age. a) The green line shows the linear fit. The shaded areas show the 95% confidence intervals. b) Blue line: fit with fractional polynomials, orange line: fit with linear B-splines with three basis functions, pink line: fit with cubic B-splines with fifteen basis functions. For the B-spline fits, the inner knots are positioned at suitable quantiles of age. There are many different methods to extend the framework of linear predictor models in order to model non-linear associations. For example, the effect of a predictor can be modeled with a polynomial function. The fractional polynomial approach ( Royston & Sauerbrei, 2008a ) extends the simple polynomial approach; here one or several power transformations are applied to a predictor. Such transformations are global, i.e., a (fractional) polynomial function is defined over the whole range of the predictor. In contrast, spline-based approaches are local smoothers: the range of the predictor is divided into segments, a different polynomial is defined on each segment, and the polynomials (all having the same degree) are smoothly joined at the knots, i.e., the endpoints of the segments, to ensure continuity of the fitted function. Spline-based approaches include, e.g., B-splines ( De, 1978a ), restricted cubic splines ( Stone & Koo, 1985a ), P-splines ( Eilers & Marx, 1996a ), and thin-plate regression splines (including smoothing splines) ( Wood, 2003a ); see Perperoglou et al. (2019a) for an overview. Fig 1 b) again shows the NHANES example, now with a fractional polynomial fit and two different B-spline fits (linear B-splines with three basis functions and cubic B-splines with fifteen basis functions). The fitted curves have different properties: all curves reflect the aforementioned non-monotonicity, but they differ with respect to (local) extrema and the degree of wiggliness. It is not immediately clear which curve is the “best one”, nor how one should define a criterion for determining such a “best” curve. Currently, there is a lack of knowledge regarding the performance of different methods for modeling functional associations. Studies comparing different methods to each other are relatively rare ( Strasak et al., 2011a ; Binder et al., 2013a ). More systematic comparison studies are needed to better understand the properties of different methods and to derive recommendations about which methods might be preferable in which application contexts, depending on the relevant characteristics of the data being analyzed and/or the complexity of the underlying true functional forms. Typically, such studies include simulated data with known true data-generating functions (“ground truths”). As an illustrative example, suppose that we perform a simulation study which systematically compares different flexible estimation methods. We choose a true data-generating mechanism inspired by the observed association of age on BMI in the NHANES data. This scenario is depicted in Fig 2 . The black line represents the assumed ground truth curve. The fractional polynomial fit and the two B-spline fits represent the curves obtained in one simulation repetition from three methods to be compared. However, it is unclear which criteria should be used to assess their success in describing the true association of the predictor and the outcome. This illustrates a general issue: it is not clear which performance measures should be used in systematic comparison studies, i.e., which measures should be used to compare the functional forms (linear or non-linear) estimated by the different methods to the ground truth. Several performance measures have been proposed in the literature ( Govindarajulu et al., 2007a ; Binder et al., 2011a ; Buchholz et al., 2014a ), but so far we are not aware of a systematic overview and a classification of the measures. In particular, it is not clear which characteristics of the estimated functional forms are addressed by different measures and how similar certain measures might be to each other. Figure 2: Illustration of a hypothetical simulation study. The black line depicts the ground truth curve. The blue line shows a fit with fractional polynomials, the orange line a fit with linear B-splines with three basis functions, and the pink line a fit with cubic B-splines with fifteen basis functions. The three fitted lines can be interpreted as the curves obtained in one simulation repetition from three different methods that are compared in the simulation study. The similarity of the estimated curves with the ground truth curve can be