Does calibration mean what they say it means; or, the reference class problem rises again

Lily Hu · 2024

Read Does calibration mean what they say it means; or, the reference class problem rises again by Lily Hu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16769.txt.

Introduction

Lily Hu writes a philosophy paper in twenty twenty-four on a fairness test for risk scores. The test is calibration: within a group, a score should match the true rate of the outcome. Many say a calibrated score means the same thing for people in distinct groups, which guards fair treatment.

Hu argues this claim fails. Each real person sits in many groups at once, so a score can be calibrated for one group and off for another. Only a perfect predictor could be calibrated for all groups at once. She calls the slip a reference class fallacy and shows the same flaw hits other group-based fairness tests.

Catalog record

CREATORLily Hu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16769

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Does calibration mean what they say it means; or, the reference class problem rises again Lily Hu Abstract Discussions of statistical criteria for fairness commonly convey the normative significance of calibration within groups by invoking what risk scores “mean.” On the Same Meaning picture, group-calibrated scores “mean the same thing” (on average) across individuals from different groups and accordingly, guard against disparate treatment of individuals based on group membership. My contention is that calibration guarantees no such thing. Since concrete actual people belong to many groups, calibration cannot ensure the kind of consistent score interpretation that the Same Meaning picture implies matters for fairness, unless calibration is met within every group to which an individual belongs. Alas only perfect predictors may meet this bar. The Same Meaning picture thus commits a reference class fallacy by inferring from calibration within some group to the “meaning” or evidential value of an individual’s score, because they are a member of that group. The reference class answer it presumes does not only lack justification; it is very likely wrong . I then show that the reference class problem besets not just calibration but other group statistical criteria that claim a close connection to fairness. Reflecting on the origins of this oversight opens a wider lens onto the predominant methodology in algorithmic fairness based on stylized cases. §1 Introduction In the literature on algorithmic fairness, group-based statistical parity conditions take stock of how an algorithm’s predictions square with true outcomes for each subpopulation or group of interest. Picking-and-choosing, mixing-and-matching different error quantities and outcome- or prediction-defined populations across different groupings yields a slew of statistical measures, parity across which has conventionally been suggestive of a kind of equal, and thus fair, treatment and lack of parity indicating divergent, possibly unfair, treatment. Within this multidisciplinary discussion, philosophers have taken up the mantle of asking which, if any, of the statistical criteria that have been proposed in technical contributions are genuinely closely tied to fairness. Some authors deflate the significance of certain statistical conditions in favor of alternatives; [ 17 , 11 , 16 ] while others aim to shore up the status of those criteria diminished by their fellow philosophers. [ 12 , 9 ] Some contributions strive to reconcile existing fairness criteria which seem to conflict; [ 1 , 13 ] while their less optimistic peers lament the slim pickings and submit criteria of their own for consideration. [ 8 ] By and large, findings within this literature have been broadly negative: of the proposed conditions, few, it seems, are possibly necessary, and none sufficient for fairness. Still, among the candidate criteria, calibration within groups has retained a normative edge, holding on most strongly to a claim to being a true requirement for fairness. Calibration speaks only to facts about model behavior, whereas other measures are sensitive to factors presumably irrelevant to fairness such as the shape of a population’s risk distribution. While other metrics are defined by reference to particular uses of an algorithm to make decisions, calibration assesses a tool’s overall performance, capturing whether across all of its possible outputs, the model is equally well-fit (on average) to the target outcome for different subpopulations or groups (e.g., by race, sex, age). Calibration within groups is thus thought to secure for decision-makers a kind of sound and consistent interpretation of risk scores, such that they are, as Kleinberg et al. put it in their landmark paper, “justified in treating people with the same score comparably with respect to the outcome.” [ 15 ] This property of calibration within groups is often glossed in the literature as ensuring that scores “mean the same thing” for individuals of different groups. Brian Hedden, for instance, writes that risk scores that are calibrated within groups, “mean the same thing, or [] have the same evidential value, regardless of the group to which the individual belongs.” [ 11 , 216] Scores that are miscalibrated, on the other hand, “mean different things” for different groups, leaving decision-makers at risk of weighing the costs of error differently across similar individuals, thereby potentially “treating individuals differently in virtue of their group membership.” [ 11 , 225] My aim in this paper is to draw out this widespread interpretation of calibration within groups, which I call the Same Meaning picture, and show why its portrayal of calibration’s distinctive normative value and its comparative advantage over other group-based statistical criteria is false. Its central claim is that because risk scores that are calibrated within groups mean the same thing (on average) across individuals from different groups, they allow for a kind of sound and even interpretation of individuals’ scores and thereby guard against decision policies, which engage in systematic disparate treatment based on the group to which an individual belongs. My contention is that calibration, unfortunately, can guarantee no such thing—or at least, it cannot guarantee the version of the Same Meaning view that is normatively significant for the fairness of an algorithmic system. Arguments that it does commit a critical error: since concrete actual people belong to many groups, calibration cannot ensure the kind of consistent interpretation of scores that the Same Meaning picture implies matters for fairness—unless calibration is met within every group to which an individual belongs. Alas only perfect predictors meet this bar. The Same Meaning motivation for calibration thus commits a reference class fallacy by presuming that calibration within some pre-specified group speaks to the “meaning” or evidential value of a particular individual’s score vis-à-vis their “true” probability, simply because they may be classified as a member of that group. This is precisely the problem of reference classes: Which of the many group probability facts corresponding to the many groups to which an individual belongs should apply when figuring the probability of their individual case? Thus, the Same Meaning picture claims an answer to the reference class problem, but it provides no argument for that answer. What is more, the reference class answer that the Same Meaning picture assumes is, in all likelihood, wrong. Or so I will argue. Showing “meaning”-based talk about calibration to rely on a mistake undercuts a premise that undergirds much debate across disciplines about the property’s normative significance. For the Same Meaning picture of calibration is invoked not just by its proponents who take it to have distinctive normative import compared with competitor criteria but also by calibration’s detractors. It is propagated not only by philosophers and legal scholars looking to convey its relevance for fairness; it is also expressed in foundational technical texts in algorithmic fairness. But it is not this paper’s primary concern to debunk calibration within groups. The story about calibration serves as an entry point into a broader investigation of the debate on statistical criteria of fairness. Identifying the mistake within commentary about calibration draws the curtain to reveal a fractal pattern, where the reference class problem is peppered throughout discussions of fairness, rearing its head in other motivations given for the normative significance of group statistical facts. As I will show, it afflicts all attempts to connect group-based statistical criteria with a notion of fairness that speaks to individuals. Furthermore, reflecting on the origins of this error opens a

Read the complete text