Synthetic-Powered Predictive Inference
Meshi Bashari ∗, Roy Maor Lotan ∗, Yonghoon Lee ∗, Edgar Dobriban, Yaniv Romano · 2025
Read Synthetic-Powered Predictive Inference by Meshi Bashari ∗, Roy Maor Lotan ∗, Yonghoon Lee ∗, Edgar Dobriban, Yaniv Romano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13432.txt.
Introduction
Forecasts often come as sets of values that should contain the true answer a set share of the time. Conformal prediction gives such sets with a firm promise, but the sets grow vague when real data are scarce. A 2025 paper by Meshi Bashari and four co-authors adds synthetic data, made by a generative model, to sharpen the sets.
Their method, called synthetic-powered predictive inference, rests on a score transporter. That tool maps the mismatch scores of real data onto those of synthetic data with a quantile match. Woven into the calibration step, it keeps the coverage promise while making no claims about how either dataset is spread. When the two score sets line up well, the forecast sets shrink a lot. Tests on image sorting and table data bear that out.
Catalog record
| CREATOR | Meshi Bashari ∗, Roy Maor Lotan ∗, Yonghoon Lee ∗, Edgar Dobriban, Yaniv Romano |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 117 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13432
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Synthetic-Powered Predictive Inference Meshi Bashari ∗, Roy Maor Lotan ∗, Yonghoon Lee ∗, Edgar Dobriban, Yaniv Romano Abstract Conformal prediction is a framework for predictive inference with a distribution-free, finite-sample guarantee. However, it tends to provide uninformative prediction sets when calibration data are scarce. This paper introduces Synthetic-powered predictive inference ( SPI ), a novel framework that incorporates synthetic data—e.g., from a generative model—to improve sample efficiency. At the core of our method is a score transporter: an empirical quantile mapping that aligns nonconformity scores from trusted, real data with those from synthetic data. By carefully integrating the score transporter into the calibration process, SPI provably achieves finite-sample coverage guarantees without making any assumptions about the real and synthetic data distributions. When the score distributions are well aligned, SPI yields substantially tighter and more informative prediction sets than standard conformal prediction. Experiments on image classification—augmenting data with synthetic diffusion-model generated images—and on tabular regression demonstrate notable improvements in predictive efficiency in data-scarce settings. 1 1 footnotetext: Equal contribution. 1 Introduction 1.1 Background and motivation Conformal prediction [ 56 , 47 , 58 ] is a general framework for quantifying predictive uncertainty, providing finite-sample statistical guarantees for any machine learning model. Given a test instance with an unknown label (e.g., an image), conformal prediction constructs a prediction set—a collection of plausible labels guaranteed to include the true label with a user-specified coverage probability (e.g., 95%). To do so, it relies on a labeled holdout calibration set to compute nonconformity scores, which measure how well a model’s prediction aligns with the true labeled outcome. These scores are then used to assess uncertainty in future predictions. Crucially, the coverage guarantee holds whenever the calibration and test data are exchangeable (e.g., i.i.d.), without any assumption on the sampling distribution. While conformal prediction offers a powerful coverage guarantee, its reliance on a holdout set limits its effectiveness when labeled data is scarce—becoming unstable and highly variable in coverage, or overly conservative and uninformative. As a result, it offers limited value in applications where labeled data is inherently limited, such as those requiring personalization or subgroup-specific guarantees. Importantly, this is not merely an abstract concern [ 7 ] —for example, in medical settings, it is natural to seek valid inference tailored to specific patient characteristics such as age, health condition, and/or other group identifiers of interest, see e.g., [ 37 , 12 ] . Similarly, in image classification tasks, one may wish to ensure that coverage holds for the true class label, see e.g., [ 57 ] . In these cases and many others, we often have only a few representative holdout examples for each group or class, which severely restricts the applicability of standard conformal prediction. Meanwhile, we are witnessing rapid progress in the ability to train accurate machine learning models even under data-scarce settings, driven by the rising quality of synthetic data produced by modern generative models and by advances in domain adaptation, see e.g., [ 10 ] . These developments inspire the question we pursue in this work: Can we rigorously enhance the sample efficiency of conformal prediction by leveraging a large pool of synthetic data—such as labeled datapoints from related subpopulations, or even data sampled from generative models? At first glance, it may appear hard to use synthetic data to boost sample efficiency in a statistically valid way. After all, the distribution of synthetic data can be completely different from that of the data of interest. 1 1 1 We refer to the limited dataset of interest as the real calibration set , to distinguish it from the (potentially synthetic) calibration data. Overcoming this challenge, we propose a principled framework that unlocks conformal prediction with the ability to incorporate synthetic data while preserving rigorous, model-agnostic, non-asymptotic coverage guarantees. Crucially, our method— SPI —provides a coverage bound that requires no assumptions about the similarity between the real and synthetic data distributions. Still, when the distribution of synthetic and real scores is close, our approach yields a substantial boost in sample efficiency—resulting in more informative prediction sets than those produced by standard conformal prediction. A discussion of related literature is deferred to Appendix B . 1.2 Preview of the proposed method and our key contributions Our key innovation is the introduction of the score transporter : a data-driven empirical quantile mapping function that transports the real calibration scores to resemble the synthetic scores. This mapping enables the construction of prediction sets for new test datapoints, leveraging the abundance of synthetic data. Crucially, the score transporter does not require data splitting, allowing full use of the real and synthetic calibration data. Furthermore, we develop a computationally efficient algorithm with a runtime complexity similar to that of standard conformal prediction. A pictorial illustration of our proposed calibration framework is provided in Figure 1 . Figure 1 : A high-level overview of the proposed method . The approach leverages a small labeled real dataset alongside a large labeled synthetic dataset. The score transporter maps scores from the real domain to the synthetic one. Calibration is then performed using the transported real scores and the synthetic scores. We support our proposed SPI framework with two theoretical guarantees. The first shows that when the synthetic and real score distributions are close, the achieved coverage closely matches the desired level. More generally, it characterizes how the distributional shift between the real and synthetic scores, and the construction of the score transporter affect the realized coverage. The second theoretical result establishes worst-case bounds on the coverage probability, which can be directly controlled by the user. This bound allows the user to set a “guardrail,” i.e., a lower bound on the coverage probability (say 80%) that holds regardless of the distribution of the synthetic data. Remarkably, this bound holds even when the synthetic data depend on the real calibration set. This flexibility enables users to adapt or filter the synthetic data to improve efficiency, for example by selecting datapoints that resemble the real ones—all without requiring any data splitting. We demonstrate the practicality of our method on multi-class classification and regression tasks. For image classification on ImageNet, we explore two practical strategies for constructing synthetic data. The first leverages a generative model (Stable Diffusion [ 45 ] or FLUX [ 29 ] ) to generate artificial images for each class. The second uses another set of real data, drawn from a different distribution, as the synthetic data. In the regression setting, we consider tabular panel data, using past panels as synthetic data and a recent panel as the real calibration set. Across all experiments, our method shows improvements in statistical efficiency—even when the real calibration set is very small, with as few as 15 datapoints. Software for reproducing the experiments is available at https://github.com/Meshiba/spi . 2 Problem setup Consider the standard setting of a prediction problem where we have m i.i.d. (real) calibration datapoints 2 2 2 Some of our results rely on a weaker assumption than i.i.d.—namely, exchangeability of the real calibration datapoints. \smash{{(X_{i},Y_{i})}_{i}\stackrel{{\scriptstyle\textnormal{iid}}}{{\sim}}P_{X,Y}=P_{X}\times P_{Y\mid X}} , for i\in[m]:=\{1,\ldots,m\} on \mathcal{X}\times\mathcal{Y} , where each X_{i}\in\mathcal{X} represents the features and Y_{i}\in\mathcal{Y} denotes the label or outcome for the i -th datapoint. Given a new test input X_{m+1}\sim P_{X} , the task is