Contrasting Cost-Agnostic and Cost-Sensitive Losses under Limited Model Capacity via ℋ \mathcal{H} -Consistency

Jessie Finocchiaro [email protected], Sanket Shah [email protected], Milind Tambe [email protected] · 2025

Read Contrasting Cost-Agnostic and Cost-Sensitive Losses under Limited Model Capacity via ℋ \mathcal{H} -Consistency by Jessie Finocchiaro [email protected], Sanket Shah [email protected], Milind Tambe [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19522.txt.

Introduction

Jessica Finocchiaro, Sanket Shah, and Milind Tambe settle a debate in machine learning about whether training should bake in the cost of errors. Their 2025 paper compares a cost-blind objective, later adjusted by a threshold, with a cost-sensitive objective trained directly. With endless data and model capacity, the two paths match in theory.

Real models have limited capacity, and there the authors prove a strict gap. When the best decision boundary for the task differs from the best cost-blind boundary, direct cost-sensitive training wins, and no threshold search can close the gap. A simple example shows the setting is plausible. The analysis uses a consistency framework that links surrogate losses to the true decision loss. Practitioners get theory for a habit many already follow.

Catalog record

CREATORJessie Finocchiaro [email protected], Sanket Shah [email protected], Milind Tambe [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-19522

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contrasting Cost-Agnostic and Cost-Sensitive Losses under Limited Model Capacity via ℋ \mathcal{H} -Consistency Jessie Finocchiaro [email protected], Sanket Shah [email protected], Milind Tambe [email protected] Abstract There is a prevalent debate in machine learning about whether practitioners should train models to optimize a task-agnostic objective (e.g., cross entropy) or incorporate the downstream decision task into the optimization objective (e.g., weighted cross entropy). In ideal settings, like those with infinite data and infinite model capacity, the two approaches are statistically equivalent for the downstream decision task. In practice, however, incorporating the decision task into model training has been shown to empirically improve task-specific performance in certain real-world scenarios. The cause of these benefits has not been theoretically studied to date. Focusing on the setting with limited model capacity through the model class ℋ \mathcal{H} , we establish a strict performance gap between post-processing a model learned with a cost-agnostic objective (e.g., thresholding a risk score prediction) and models learned by optimizing a cost-sensitive model without a threshold search. In particular, we establish this gap when there is a hypothesis recovering the optimal decision boundary for the discrete task, but it does not align with the optimal cost-agnostic hypothesis, and give a simple example demonstrating the plausibility of this setting. With assumptions that are hard to verify in practice, we demonstrate that this gap generally exists on classification datasets from the UCI repository, particularly with very simple models. 1 Introduction In training machine learning models for discrete prediction tasks, the practitioner faces a plethora of design decisions. One of these choices is the loss function used as a training objective: should loss functions be hand-crafted for a “cost-sensitive” decision task at hand, or does it suffice to post-process a model trained on a generic loss like cross entropy? The debate about this choice is longstanding; the former camp argues that, when faced with the inevitability of imperfect prediction, optimizing a custom surrogate will yield better predictions for the task at hand ( Vapnik, 2000 ; Scott, 2012 ; Ramaswamy et al., 2016 ; Wilder et al., 2019 ; Elmachtoub & Grigas, 2022 ; Shah et al., 2022 ) . In contrast, the other camp argues that one can often adapt the model for the task at hand with clever post-processing while generalizing to unknown decision-makers ( Dawid, 1982 ; Seidenfeld, 1985 ; Buja et al., 2005 ; Reid & Williamson, 2009 ; Reid & Williamson, 2011 ; Gopalan et al., 2022 ) . As an example of such a cost-sensitive decision task, suppose the cost of a false positive is much higher than the cost of a false negative in a binary classification problem. For example, an IT team coping with phishing attacks will probably prefer to cope with false positives (with consequences of some genuine e-mails being flagged as spam) than false negatives (leading to data breaches arising from successful phishing attacks) in a spam detection system. In that case, one might have a threshold \tau\in[0,1] in mind where they want to flag an email as spam if a model predicts \Pr[ spam \mid email contents ]\geq\tau . In “nice” settings where perfect prediction is possible, it is known that the two approaches are statistically equivalent ( Frongillo & Kash, 2021 ) . However, in practice, the decision-focused learning literature has empirically improved task-specific performance by designing custom loss functions ( Elmachtoub & Grigas, 2022 ; Wilder et al., 2019 ) . We examine the gap induced by loss function choice in the setting of cost-sensitive classification from a theoretical perspective. Here, cost-sensitive classification refers to a setting where “costs” for different types of errors (e.g., false positives and negatives) are written as a cost matrix \ell . Our goal in this work is not to advocate for cost-sensitive surrogates over cost-agnostic ones generally, but rather to serve as a first step towards characterizing settings in which the choice of surrogate training loss affects the quality of decision-theoretic performance. While we show positive results for cost-sensitive surrogates, we also note that cost-agnostic losses have their place. For example, omnipredictors ( Gopalan et al., 2022 ) enable post-processing of a model’s predictions for a broad class of decision-makers, which is a benefit sacrificed by selecting a cost-sensitive training loss. We establish a gap between the performance of models trained on cost-sensitive vs agnostic losses in the setting of limited model capacity through the lens of \mathcal{H} -consistency. \mathcal{H} -consistency examines whether learning the best model in a hypothesis class \mathcal{H} with respect to a training loss L implies making the best decision for a cost-sensitive target problem \ell . We show that, for “small enough” model classes \mathcal{H} (relative to the data-generating distribution P\in\Delta(\mathcal{X}\times\mathcal{Y}) ), there are instances where a cost-sensitive surrogate L^{cs} is \mathcal{H} -consistent with respect to the target problem \ell while a canonical cost-agnostic surrogate L^{ag} is not \mathcal{H} -consistent, regardless of post-processing choices . Concretely, we construct a counterexample giving some intuition for our setting in Section 2 . With this counterexample, we show in Section 4 that a cost-agnostic surrogate (e.g., cross entropy, squared error) plus post-processing are not \mathcal{H} -consistent, and can fail to perform well for the cost-sensitive target. In contrast, we evaluate the \mathcal{H} -consistency of a class of surrogates called embeddings ( Finocchiaro et al., 2024 ) . These task-specific losses in Section 5 include piecewise linear losses like the weighted hinge. By “corresponding exactly to the optimal decision for \ell ,” we show these losses yield \mathcal{H} -consistency guarantees in the same setting as section 4 . In Section 6 we analyze weighted analogs of canonical cost-agnostic surrogates (e.g., Weighted Cross Entropy, Squared Error, etc.), and show that even in binary outcome settings, stronger assumptions are needed to establish \mathcal{H} -consistency. Our theoretical results establish a strict gap between the performance of small ML models trained on cost-sensitive surrogates and those trained on cost-agnostic surrogate losses, even when the cost-agnostic losses are equipped with a threshold search to post-process model predictions. Finally, in Section 7 , we empirically evaluate the existence of this performance gap when distributional assumptions cannot be verified. 2 Motivating example To develop an intuition for where a performance gap between cost-sensitive and cost-agnostic models emerges, we turn to Example 1 , which gives a cost-sensitive binary classification problem \ell^{\alpha} with costs in Table 1 and learned hypotheses visualized in Figure 5 . In essence, the target loss \ell^{\alpha} imposes cost 1-\alpha for a false negative, and cost \alpha for a false positive. A model h:\mathcal{X}\to\mathbb{R} defines contours over the space \mathcal{X} , and the possible shapes of these contours are determined by the model class \mathcal{H} . Predictions in \mathbb{R} are then post-processed into decisions in \mathcal{R}\subset\mathbb{Z} through a link function, discussed shortly. For our motivating example, we consider optimizing over the model class \mathcal{H}_{\mathrm{lin}}=\{x\mapsto\rho(w^{T}x)\mid w\in\mathbb{R}^{2}\} , where \rho:\mathbb{R}\to[-1,1] is the sign-based translation of the sigmoid 2\sigma(z)-1 . Example 1 . Consider features \mathcal{X}\subseteq\mathbb{R}^{2} and binary labels \mathcal{Y}=\{-1,1\} drawn from the distribution P such that \displaystyle Y\sim\mathbf{Bern}\left(\frac{1}{2}\right)~~~~X_{2}\sim U[0,1],~~~~X_{1}\sim\mathcal{N}(y\cdot X_{2},X_{2}^{2}). The optimal cost-agnostic linear hypothesis over P is simply h^{ag}_{{\mathrm{lin}}}(x)=\rho(x_{1}) 1 1 1 Technically, h^{ag}_{{\mathrm{lin}}}(x)=\rho(cx_{1}) for some c\neq 0 , but we leverage a degree of freedom to take c=1 . because

Read the complete text