Panprediction: Optimal Predictions for Any Downstream Task and Loss

Sivaraman Balakrishnan, Nika Haghtalab, Daniel Hsu, Brian Lee, Eric Zhao · 2025

Read Panprediction: Optimal Predictions for Any Downstream Task and Loss by Sivaraman Balakrishnan, Nika Haghtalab, Daniel Hsu, Brian Lee, Eric Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27638.txt.

Introduction

Sivaraman Balakrishnan, Nika Haghtalab, Daniel Hsu, Brian Lee, and Eric Zhao formalize panprediction in 2025, a way to train one model for many future tasks and losses at once. Classic training fixes one task and one loss, which is a score for error. New practice instead asks one model to hold enough signal to serve many later uses.

Panprediction extends two prior ideas that each covered only half the goal. The team gives methods that learn such models from sample counts close to the single-task case. Both fixed and random forecast rules are covered under mild conditions. The finding is striking: serving endless tasks and losses can cost little more than serving one.

Catalog record

CREATORSivaraman Balakrishnan, Nika Haghtalab, Daniel Hsu, Brian Lee, Eric Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Panprediction: Optimal Predictions for Any Downstream Task and Loss Sivaraman Balakrishnan, Nika Haghtalab, Daniel Hsu, Brian Lee, Eric Zhao Abstract Supervised learning is classically formulated as training a model to minimize a fixed loss function over a fixed distribution, or task. However, an emerging paradigm instead views model training as extracting enough information from data so that the model can be used to minimize many losses on many downstream tasks. We formalize a mathematical framework for this paradigm, which we call panprediction , and study its statistical complexity. Formally, panprediction generalizes omniprediction and sits upstream from multi-group learning, which respectively focus on predictions that generalize to many downstream losses or many downstream tasks, but not both. Concretely, we design algorithms that learn deterministic and randomized panpredictors with O ~ ​ ( 1 / ε 3 ) \widetilde{O}(1/\varepsilon^{3}) and O ~ ​ ( 1 / ε 2 ) \widetilde{O}(1/\varepsilon^{2}) samples, respectively. Our results demonstrate that under mild assumptions, simultaneously minimizing infinitely many losses on infinitely many tasks can be as statistically easy as minimizing one loss on one task. Along the way, we improve the best known sample complexity guarantee of deterministic omniprediction by a factor of 1 / ε 1/\varepsilon , and match all other known sample complexity guarantees of omniprediction and multi-group learning. Our key technical ingredient is a nearly lossless reduction from panprediction to a statistically efficient notion of calibration, called step calibration . 1 Introduction Consider the problem of predicting the probability of an adverse medical event for a patient based on their health records—be it in-hospital mortality over 12 hours, re-admission over 30 days, or a cardiovascular event over 10 years. A range of decision makers across the healthcare system—ICU doctors, discharge planners, actuaries, and more—wish to incorporate these probabilities into their decisions. However, this problem is complicated by the fact that each decision maker defines prediction quality differently: an ICU doctor who must rapidly take action may care most about the zero-one loss, while an actuary who seeks precise probability estimates may care most about the square loss. Further complicating the problem is how each decision maker focuses on different, possibly overlapping subgroups of patients: those with specific pre-existing conditions, those in a certain age group, and more. What does it mean for a single predictor to be “good” for such a heterogeneous population of decision makers? How can such a predictor be learned? \bigl(\mathcal{G},\mathcal{H},\varepsilon\bigr) -Step Calibration Deterministic \widetilde{O}\left(\varepsilon^{-3}\right) Randomized \widetilde{O}\left(\varepsilon^{-2}\right) \bigl(\mathcal{L},\mathcal{G},\mathcal{H},\varepsilon\bigr) -Panprediction Deterministic \widetilde{O}\left(\varepsilon^{-3}\right) Randomized \widetilde{O}\left(\varepsilon^{-2}\right) (\ell,\mathcal{G},\mathcal{H},\varepsilon) -Multi-group Learning Match best known det. and rand. sample complexity [ 1 ] \bigl(\mathcal{L},\mathcal{H},\varepsilon\bigr) -Omniprediction Improve det. sample complexity by \varepsilon^{-1} , match rand. sample complexity [ 2 ] Figure 1: Panprediction is a nearly lossless generalization of omniprediction and multi-group learning in the sense that sample complexity guarantees are preserved (up to logarithmic factors), or improved. Similarly, the reduction from panprediction to step calibration preserves sample complexity guarantees. From one perspective, we have simply described a collection of binary prediction problems, each defined by a loss function and subgroup, or task, of interest. Classic supervised learning techniques can readily produce a good model for each problem. However, this approach requires that the number of samples needed to produce good predictions for all decision makers scales linearly in the number of losses and subgroups of interest, which can be prohibitively large. In this work, we introduce panprediction , an alternative framework for constructing a single probability predictor that any decision maker—focused on any loss and any subgroup—can easily post-process to solve their specialized problem. Importantly, panpredictors guarantee that decision makers could not have made better predictions by training a bespoke model for their problem. Moreover, the sample complexity of panprediction scales only logarithmically in the number of basis functions needed to linearly approximate losses of interest, and also in the number of subgroups. Formally, we say a probability predictor p^{\star}:\mathcal{X}\to[0,1] is a (\mathcal{L},\mathcal{G},\mathcal{H},\varepsilon) -panpredictor if for every loss \ell\in\mathcal{L} and group g\in\mathcal{G} , the predictor p^{\star} can be post-processed into a predictor h^{\star} that is \varepsilon -optimal, meaning \displaystyle\mathbb{E}_{(x,y)\sim D}\bigl[\ell\bigl(h^{*}(x),y\bigr)\bigm|g(x)=1\bigr]\approx_{\varepsilon}\min_{h\in\mathcal{H}}\mathbb{E}_{(x,y)\sim D}\bigl[\ell\bigl(h(x),y\bigr)\bigm|g(x)=1\bigr], where \mathcal{H} is a competitor hypothesis class. Supposing for simplicity that \mathcal{G} and \mathcal{H} have finite cardinality, our proposed algorithms learn deterministic panpredictors with \widetilde{O}(\varepsilon^{-3}\log(|\mathcal{G}||\mathcal{H}|)) samples, and randomized panpredictors with \widetilde{O}(\varepsilon^{-2}\log(|\mathcal{G}||\mathcal{H}|)) samples. Notably, our sample complexity guarantees account for \mathcal{L} through a \log(1/\varepsilon) factor, with no dependence on the cardinality or covering number of \mathcal{L} . Our results are enabled by the observation that panprediction can be reduced to step calibration [ 3 ] , a statistically efficient notion of calibration also known as proper calibration [ 2 ] . This answers our first question: a single predictor is “good” for a heterogeneous array of losses and subgroups if it is step calibrated. Algorithmically, we construct step calibrated predictors using tools from multi-objective learning, a powerful framework recently developed to tighten the sample complexity of various calibration notions [ 4 , 5 ] . This answers our second question on how such “good” predictors can be learned. Panprediction builds on recent advances in machine learning theory that seek models with greater adaptability to many downstream losses or tasks. Omniprediction [ 6 ] studies predictors whose outputs can be post-processed to perform well according to many loss functions, but for a single fixed distribution, or task. Conversely, multi-group learning [ 7 , 1 ] ensures robust performance across many downstream subgroups, or tasks, but for a single fixed loss function. We generalize both lines of work and show that simultaneous adaptability to many downstream losses and tasks is readily attainable. Figure 1 illustrates the relationship between these concepts. Our approach yields several benefits over prior works. Quantitatively, we improve the sample complexity of deterministic omniprediction by a factor of 1/\varepsilon , and match all other known sample complexities of omniprediction and multi-group learning. Qualitatively, we provide a unified algorithmic path to omniprediction and multi-group learning—which prior works have treated with bespoke approaches that had difficulty learning, e.g., deterministic predictors—and significantly simplify proofs. 1.1 Related Works Omniprediction Gopalan et al. [ 6 ] initiated the study of omniprediction by showing that an appropriately calibrated predictor can be post-processed to minimize a range of convex and Lipschitz losses. Subsequent work deepened the conceptual connection between omniprediction and calibration by formalizing the notion of a predictor being indistinguishable from the Bayes predictor, as measured by a set of loss functions [ 8 ] . Using these insights, recent work improved the sample complexity and oracle efficiency of omniprediction by sidestepping full calibration and relying on statistically efficient notions of calibration [ 2 ] . We carefully extend this line of work to account for the more challenging setting where loss functions are evaluated over overlapping subgroups of the domain. Multi-group learning Motivated by subgroup fairness, Blum and Lykouris [ 9 ] and Rothblum and Yona [ 7 ] respectively initiated the study of multi-group learning in the sequential and batch settings. A rich line of work has since refined sample complexity guarantees [ 1 ] and studied variants with oracle-efficiency [ 10 ] , hierarchical group structure [ 11 ] , and robustness concerns [ 12 ] . Typical multi-group algorithms leverage reductions to the sleeping experts problem. In contrast,

Read the complete text