On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift

Alejandro Moreo · 2025

Read On the Interconnections of Calibration, Quantification, and Classifier Accuracy Prediction under Dataset Shift by Alejandro Moreo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11380.txt.

Introduction

Alejandro Moreo, writing alone in 2025, proves that three seemingly separate problems in machine learning are secretly the same problem. The setting is dataset shift, where test data drifts from training data. The three tasks are calibrating a classifier's scores, estimating the share of positives in new data, and predicting the classifier's accuracy on that new data.

Moreo shows each task can be reduced to the others: an oracle for any one solves all three. That equivalence lets methods cross borders. Adapting a strong calibration method yields a quantification method, and so on. In tests, the borrowed methods often matched or beat the dedicated ones. The paper urges the three research communities to build unified tools.

Catalog record

CREATORAlejandro Moreo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE145 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11380

Browse the full library · About the Tower of Babel