Image Super-Resolution with Guarantees via Conformal Generative Models

Eduardo Adame, Daniel Csillag, Guilherme Tegoni Goedert · 2025

Read Image Super-Resolution with Guarantees via Conformal Generative Models by Eduardo Adame, Daniel Csillag, Guilherme Tegoni Goedert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09664.txt.

Introduction

Eduardo Adame, Daniel Csillag, and Guilherme Tegoni Goedert add a trust map to AI image upscaling, in a 2025 paper. Generative models can invent detail when they enlarge a photo, and users cannot tell which parts are safe to believe. Their method paints a confidence mask over the result, marking where the output matches the input closely enough to trust.

The mask is built with conformal prediction, a statistics toolkit that comes with guarantees. It works with any generator, even one hidden behind an API, and needs only easy-to-get calibration data. The authors prove bounds on error and quality, then test the masks in practice. The aim is restoration that says, plainly, where it might be guessing.

Catalog record

CREATOREduardo Adame, Daniel Csillag, Guilherme Tegoni Goedert
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09664

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Image Super-Resolution with Guarantees via Conformal Generative Models Eduardo Adame, Daniel Csillag, Guilherme Tegoni Goedert Abstract The increasing use of generative ML foundation models for image super-resolution calls for robust and interpretable uncertainty quantification methods. We address this need by presenting a novel approach based on conformal prediction techniques to create a ‘confidence mask’ capable of reliably and intuitively communicating where the generated image can be trusted. Our method is adaptable to any black-box generative model, including those locked behind an opaque API, requires only easily attainable data for calibration, and is highly customizable via the choice of a local image similarity metric. We prove strong theoretical guarantees for our method that span fidelity error control (according to our local image similarity metric), reconstruction quality, and robustness in the face of data leakage. Finally, we empirically evaluate these results and establish our method’s solid performance. † † affiliationnotice: Equal contribution 1 Introduction Generative ML foundation models led to massive leaps in the capabilities of modern image synthesis and processing, spanning domains such as image generation, inpainting, and super-resolution. Particularly in the case of image super-resolution, recent methods have become considerably adept at leveraging patterns in images to better recover complex textures, geometries, lighting and more. In testament to these improvements, leading manufacturers are constantly improving and deploying tools based on these frameworks in every new generation of consumer devices. These widespread applications highlight an important question: How trustworthy are the predictions of these models? When a model does some particular inpainting or super-resolution infill, what guarantees do we have that its predictions are truly accurate to reality, and not mere hallucinations? Therefore, it would be most desirable to have a proper uncertainty quantification over the predicted image. However, most of the previous contributions to this endeavor have suffered from a lack of interpretability. In order to be widely adopted, a new framework should clearly communicate its uncertainty estimates to the public in a way that reflects how they will be used. This demand for interpretability fundamentally guides the properties and theoretical guarantees we seek to establish for our predictions, and thus on the underlying procedure. Of course, all of this is compounded by the usual challenges of having to do trustworthy uncertainty quantification that is model agnostic and can be employed effectively atop any “black-box” foundation model. In this paper, we address all these issues by proposing a method based on techniques from conformal prediction ( Vovk et al. 2005 ) and conformal risk control ( Angelopoulos et al. 2022a ) , while employing metrics designed for interpretability in concrete applications. All our method requires is a handful of unlabelled high-resolution images that were not present in the training set for the diffusion model, and we achieve strong guarantees on our predictions that are also intuitive to the user and robust to violations of our key assumptions. Our main contributions are: • A new method to quantify uncertainty in images inpainted or augmented by diffusion-based models. Our method can work atop any black-box diffusion model, including models that are locked behind an opaque API, and requires only easily-attainable unlabelled data for calibration. • We identify additional theoretical guarantees enjoyed by our model. In particular, we prove that our method also controls the PSNR of the predicted images, and show that it is reasonably robust to data leakage, reinforcing the effectiveness and robustness of our method. • A comprehensive study of modelling choices in our approach, revealing particular modifications from the base procedure that can significantly enhance performance. Particularly, we demonstrate that certain applications of low-pass filters can greatly improve our method’s effectiveness. Related work There have been some attempts on uncertainty quantification for image super-resolution with varying form. But most existing works, e.g., ( Teneggi et al. 2023 ) , ( Angelopoulos et al. 2022b ) , have as their goal to produce, rather than a single image, an interval-valued image (i.e., image where each pixel is represented by an interval rather than a single value), and ensure that these intervals will, with high probability, contain the ‘true’ pixel values. However, this is fairly non intuitive for the user and its underlying guarantees are a bit lax. As far as we are aware, the only existing solution that proposes some other way of quantifying uncertainty is ( Kutiel et al. 2023 ) , which produces a mask over the predicted image indicating the trustworthy region, much like us. However, their solution has no probabilistic guarantee, which is fundamental for reliable uncertainty quantification. Ours, in contrast, is backed by a plethora of such theorems. Finally, our proposed solution is strongly related to the existing applications of conformal prediction to semantic segmentation, e.g., ( Csillag et al. 2023 ; Mossina et al. 2024 ) . 2 Method 2.1 Conformal Mask Calibration Let us be supplied with any generative image super-resolution model \mu:[0,1]^{w\times h\times 3}\to[0,1]^{kw\times kh\times 3} , where w and h stand for the low-resolution image dimensions, and k is the upscaling factor. Naturally, this is a stochastic function due to the generative nature of the model, so an intuitive (albeit nonrigorous) way to quantify model indecision would be to aggregate many realizations of the output image (e.g. by computing the variance of generated pixels). However, our methodology is capable of working naturally when supplied with an arbitrary estimate of model indecision that can be described by a function \sigma:[0,1]^{w\times h\times 3}\to\mathbb{R}^{kw\times kh} . We further discuss useful constructions of \sigma in Section 2.2 . Having received a lower resolution image X , we must consider how the model prediction \widehat{Y}=\mu(X) differs from the true high resolution image Y . Our goal is to find a “confidence mask” M(X) that indicates the region of the image whose content we trust, i.e. the pixels in the predicted image with indecision below a sought threshold. Formally, the mask is a (possibly stochastic) function M:[0,1]^{w\times h\times 3}\to\{0,1\}^{kw\times kh} that has image M(X)=\left\{p\in\mathbb{Z}\times\mathbb{Z}:[\sigma(X)]_{p}\leq t\right\} , where [\bullet]_{p} is the image value at pixel p (be it binary, grayscale or colored) and t is a desired indecision threshold. We seek masks that satisfy fidelity guarantees between the true and predicted high-resolution images. This fidelity is measured by a fidelity error defined as \sup_{p\in M}D_{p}(Y,\widehat{Y}) , where D_{p} is a function that measures the difference between two images around some pixel location p . We can employ any local measure of the difference as long as 0\leq D_{p}(Y,\widehat{Y})\leq 3 for all p , Y and \widehat{Y} (e.g. D_{p}(Y,\widehat{Y})=\|[Y]_{p}-[\widehat{Y}]_{p}\|_{1} ). There are many different useful choices for D_{p} , with a few explored in Section 2.3 . Equipped with the previous definitions, we are able to produce these fidelity masks M_{\alpha}(X) for any desired fidelity level \alpha\in[0,1] with the guarantee that \mathbb{E}\left[\sup_{p\in M_{\alpha}(X)}D_{p}\left(\mu(X),Y\right)\right]\leq\alpha by thresholding the output of \sigma by some parameter t in the construction of the mask. This parameter can then be calibrated for by using techniques of conformal prediction ( Vovk et al. 2005 ) and conformal risk control ( Angelopoulos et al. 2022a ) with just access to unlabelled hold-out data, which has not been used to obtain either \mu or \sigma (if there is data contamination, weaker guarantees hold; see Proposition 3.2 ). In

Read the complete text