Stop Chasing the C-index: This Is How We Should Evaluate Our Survival Models
Christian Marius Lillelund, Shi-ang Qi, Russell Greiner, Christian Fischer Pedersen · 2025
Read Stop Chasing the C-index: This Is How We Should Evaluate Our Survival Models by Christian Marius Lillelund, Shi-ang Qi, Russell Greiner, Christian Fischer Pedersen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02075.txt.
Introduction
Lillelund, Qi, Greiner, and Pedersen wrote in twenty twenty-five a position paper against overusing one score for survival models. Survival models predict how long until an event, like death or failure. The C-index is a popular score for ranking such predictions. The authors argue it is often used in ways that clash with the model's actual goal. Hidden assumptions about censored data, where follow-up ends early, add to the problem.
They list what a good evaluation should require. Their double-helix ladder picture demands that metric and assumptions align. Controlled experiments show that misalignment can flip which model looks better. The paper ends with practical guidance for fair evaluation. It asks the field to stop chasing a single number.
Catalog record
| CREATOR | Christian Marius Lillelund, Shi-ang Qi, Russell Greiner, Christian Fischer Pedersen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02075
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stop Chasing the C-index: This Is How We Should Evaluate Our Survival Models Christian Marius Lillelund, Shi-ang Qi, Russell Greiner, Christian Fischer Pedersen Abstract We argue that many survival analysis and time-to-event models are incorrectly evaluated. First, we survey many examples of evaluation approaches in the literature and find that most rely on concordance (C-index). However, the C-index only measures a model’s discriminative ability and does not assess other important aspects, such as the accuracy of the time-to-event predictions or the calibration of the model’s probabilistic estimates. Next, we present a set of key desiderata for choosing the right evaluation metric and discuss their pros and cons. These are tailored to the challenges in survival analysis, such as sensitivity to miscalibration and various censoring assumptions. We hypothesize that the current development of survival metrics conforms to a double-helix ladder, and that model validity and metric validity must stand on the same rung of the assumption ladder. Finally, we discuss the appropriate methods for evaluating a survival model in practice and summarize various viewpoints opposing our analysis. 1 Introduction Time-to-event outcomes have an extensive history in research, where the focus was initially to predict the time until an event of interest – such as death, relapse, or recovery in medicine, service cancellation in business, or failure of a mechanical component in engineering – for a particular instance. Much of the pioneering research on time-to-event outcomes focused on time until death, which led to the field being commonly known as survival analysis [ 1 ] . Unlike standard regression tasks, a key challenge in training and evaluating survival models lies in handling censored data – instances where the event of interest has not been observed within the follow-up period [ 2 ] . In a medical study, only some of the patients may experience the event; the others may be lost to follow-up, withdraw from the study, or remain alive at the end of the study. For these patients, we do not know the exact event time, but instead know only a lower bound of that event time. This is known as right-censoring . Even though the task resembles regression, few learned models explicitly predict the time until the event, for a novel instance. Instead, many models M(\cdot) predict only a risk score, M(x)\in\mathbb{R} for each instance encoded as x\in X (typically \mathbb{R}^{d} ), with the understanding that M(z)>M(y) means M(\cdot) is predicting that z will die before y . This naturally leads to the concordance measure (aka C-index or simply CI), which (in essence) is measuring how often this ranking is correct –- i.e. , of the times when M(z)>M(y) , how often did z actually die before y . Note that there are many other types of survival models (see [ 3 ] ), including individual survival distribution (ISD) models, which for each x , estimates \{S(t\mid x)\mid t\geq 0\} , where S(t\mid x) is the probability that the target event will occur for x at a time T later than t [ 4 ] . We can view this model as producing a risk of an instance, by setting an instance’s risk to be (the negative of) the mean of that instance’s distribution. This means that we can use a concordance measure to evaluate such models. But these ISDs provide more than just risk scores -– they can also compute the probability of surviving until a specific time (say, 1 year), the median survival time, or the probability of experiencing the event within a specific time window, etc. With this much flexibility, we are not constrained to only evaluate the model with concordance of the predictions. However, to this day, the research community has focused primarily on optimizing the C-index – for models that can produce a risk score, including simple “risk models” , as well as models that generate ISDs, etc. (see Figure 1 ). In a recent survey, Zhou et al. 2023 report that over 80% of survival analysis studies published in leading statistical journals in 2023 use the C-index as their primary evaluation metric. At the same time, the metric has been criticized from a statistical and clinical perspective [ 6 , 7 , 8 , 9 ] . Cook 2007 shows that the C-index is insensitive to the addition of new covariates in a model, even if the new covariates are statistically and clinically significant. In low-risk populations, the C-index often compares patients who have very similar risk probabilities, which may not give meaningful insights. For example, consider two very healthy patients, A and B. Patient A dies after 30 years and patient B after 31 years. Although they are considered “concordant” by the C-index, this concordance offers little practical value to medical professionals because their risk profiles are nearly identical. Vickers and Cronin 2010 argue that because the C-index depends only on the ranks of the predicted values, models with inaccurate survival predictions can have C-indices that are much higher than those of a competing model with more accurate predictions. Halligan et al. 2015 argue that the concepts of sensitivity and specificity can be meaningful in isolation, but since the C-index combines them, it loses meaning. Position. In this paper, we encourage researchers and practitioners in survival analysis, when dealing with models, to move beyond the narrow focus on discriminative performance and adopt a broader, more comprehensive evaluation strategy. We suggest first identifying the task at hand and then deciding on which evaluation metric to use that matches that task. Although previous work has highlighted why and when the C-index is problematic [ 6 , 7 , 8 , 9 ] , this paper is the first to introduce concrete desiderata for survival evaluation and provide a guide for selecting the appropriate metric for different evaluation needs. Since no single metric is appropriate in all scenarios, we recommend considering the research objective, the assumptions about the data and these proposed desiderata, and then deciding on a metric. Furthermore, in settings where assumptions like random censoring do not hold, evaluation strategies should be adapted accordingly, rather than defaulting to the simple C-index. Lastly, we argue that researchers should consider designing new evaluation metrics, to address the specific task, for when that task does not follow the commonly used assumptions. Overview. Section 2 presents a formal definition of survival analysis and common censoring assumptions. Section 3 outlines a structured approach to evaluating models, centered on five key desiderata. Section 4 introduces three evaluation aspects and reviews commonly used metrics. Section 5 introduces a guiding principle for model evaluation, and Section 6 presents practical examples, where the metric does not match the use. Section 7 is a practical guide on how to evaluate survival models, and Section 8 provides some alternative views to our analysis and concludes the paper. Figure 1: Summary of the evaluation metrics used in recent survival analysis publications (2023-2025), based on an approximate count of accepted papers in archival venues whose abstracts and methods sections contain relevant keywords (see Appendix A ). Most papers rely on metrics that only assess the model’s ability to rank risk scores, while fewer use metrics that evaluate the time-to-event estimates or how well the survival probabilities match