Conf-Gen: Conformal Uncertainty Quantification for Generative Models
Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui, Marc T. Law, Kin Kwan Leung · 2026
Read Conf-Gen: Conformal Uncertainty Quantification for Generative Models by Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui, Marc T. Law, Kin Kwan Leung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28920.txt.
Introduction
Gabriel Loaiza-Ganem and colleagues extend conformal risk control to generative models with a framework called Conf-Gen. Conformal prediction gives formal uncertainty guarantees for supervised learning. Image and language generators fall outside its usual scope, so Conf-Gen adapts the method while easing its assumptions. The framework joins past attempts to apply conformal ideas to language models and opens new uses. These include guarantees on non-memorized images, enough clarifying questions from a chat system, and correct agent output.
Catalog record
| CREATOR | Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui, Marc T. Law, Kin Kwan Leung |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 112772 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Conf-Gen: Conformal Uncertainty Quantification for Generative Models Gabriel Loaiza-Ganem, Kevin Zhang, Wei Cui, Marc T. Law, Kin Kwan Leung Abstract Conformal prediction (CP) and its extension, conformal risk control (CRC), are established frameworks for quantifying uncertainty in supervised machine learning through formal guarantees. However, recent breakthroughs in artificial intelligence (AI) have been driven by unsupervised generative models, such as large language models (LLMs) and image generators, which are not directly compatible with CP or CRC. In this work we introduce conformal generation (Conf-Gen), a general framework adapting CRC to generative tasks while relaxing its theoretical assumptions. Conf-Gen unifies and generalizes previous attempts to apply CP to LLMs, and extends conformal methodology to entirely new domains. We demonstrate the flexibility of Conf-Gen through some novel applications, including obtaining conformal guarantees on: image generators producing non-memorized images, conversational AI systems having asked enough clarifying questions, and the output of AI agents being correct. Section 1 Introduction Significant empirical breakthroughs have been achieved in artificial intelligence (AI) in recent years, from the advent of large language models ( Brown et al. 2020 ; OpenAI 2024 , LLMs;) , to realistic image generators ( Ramesh et al. 2022 ; Rombach et al. 2022 ) , agentic systems ( Park et al. 2023 ) , and more ( Silver et al. 2016 ; Potapenko et al. 2021 ) . Most of these developments have moved away from supervised learning and instead leverage generative models. Despite these strides, uncertainty quantification (UQ) remains a critical challenge in AI, particularly for generative models, thereby hindering their deployment in high-stakes applications such as healthcare ( Begoli et al. 2019 ) and scientific discovery ( Wang et al. 2023a ) . Conformal prediction ( Gammerman et al. 1998 ; Vovk et al. 1999 ; Saunders et al. 1999 ; Gammerman & Vovk 2007 , CP;) is a principled approach for UQ in supervised learning. CP quantifies uncertainty by using features X to construct a set \mathcal{C}(X) of predictions guaranteed to contain the ground truth Y_{\text{GT}} with a user-specified probability. Conformal risk control ( Angelopoulos et al. 2024b , CRC;) extends CP by lower-bounding the expectation of a more general utility function, U(\mathcal{C}(X),Y_{\text{GT}}) , of which CP is a special case. Because conformal guarantees are distribution-free, they are particularly appealing as an approach for UQ capable of handling the complex distributions represented by generative models. Yet, despite its generality, CRC is not directly applicable to generative tasks. As a consequence, work leveraging conformal ideas in the context of generative models tends to be highly task-specific. Figure 1: Simplified illustration of our method. The selection function \mathbf{C}_{\lambda}(x,\mathbf{y}) processes an input x and a sequence of generations \mathbf{y} to yield an output that becomes increasingly conservative as \lambda increases. Conf-Gen identifies a value \hat{\lambda} that formally ensures the output of \mathbf{C}_{\hat{\lambda}} satisfies a target admissibility requirement in expectation. Top : when \mathbf{y} consists of i.i.d. responses to a query x , admissibility ensures the output contains at least one correct answer. Bottom : when the images in \mathbf{y} are generated by progressively modifying the prompt x , admissibility ensures the outputted image is not memorized. In this work we make the following contributions: (a) Framework. We propose conformal generation (Conf-Gen, depicted in Figure 1 ), a framework extending CRC for rigorous UQ in generative modelling. Conf-Gen can be applied to structures beyond sets, such as sequences, and formally lower bounds the expected admissibility (the analogue of U for generative tasks) of the selected structures. We show that existing task-specific conformal methods can be recovered as special cases of Conf-Gen. (b) Theory. Conf-Gen relaxes the theoretical assumptions of CRC. Most importantly, we weaken a monotonicity assumption and we derive a more general reverse (upper) bound on the expected utility/admissibility. (c) Python Package. Naïvely carrying out the computations required by Conf-Gen is sometimes intractable. We identify patterns in the design space of Conf-Gen components which enable efficient computations, and provide a flexible Python package ( https://github.com/layer6ai-labs/conf-gen ) supporting the full range of Conf-Gen applications. (d) Empirical Validation. We demonstrate that Conf-Gen outperforms state-of-the-art conformal baselines in LLM question answering and showcase its versatility across various novel tasks. Specifically, we show conformal guarantees for: generating non-memorized images; ensuring a conversational LLM has asked enough clarifying questions; producing a sequence of agentic AI outputs containing a correct solution; and selecting a subset of trees from a random forest ( Breiman 2001 ) containing enough trees making the correct prediction. Section 2 Conformal Prediction and Risk Control Split Conformal Prediction. We present a short summary of key concepts from CP that are relevant to Conf-Gen; for more thorough presentations, we refer the reader to Vovk et al. 2005 , Angelopoulos & Bates 2021 , and Angelopoulos et al. 2024a . We focus on split conformal prediction ( Papadopoulos et al. 2002 ) , a particular instance of CP; henceforth, we will use the abbreviation CP to refer to split conformal prediction. In CP, we assume access to a calibration dataset containing features X^{(i)}\in\mathcal{X} and their corresponding ground truth target Y^{(i)}_{\text{GT}}\in\mathcal{Y}_{\text{GT}} for i=1,\dots,n . Then, given a new X^{(n+1)} , the goal of CP is to use the calibration dataset to produce a set \mathcal{C}(X^{(n+1)})\subseteq\mathcal{Y}_{\text{GT}} which contains Y^{(n+1)}_{\text{GT}} with probability at least \gamma , where \gamma is a user-specified quantity. To achieve this goal, CP uses a score function S^{\downarrow}:\mathcal{X}\times\mathcal{Y}_{\text{GT}}\rightarrow\mathbb{R} , with large values of S^{\downarrow}(x,y) conveying low confidence that y is compatible with x . For example, in a classification setting, we could have S^{\downarrow}(x,y)=1-S^{\uparrow}(x,y) , where S^{\uparrow}(x,y) is the probability assigned by a pre-trained classifier to y when given x as input. For a given \lambda\in\mathbb{R} , CP considers sets of the form: \mathcal{C}_{\lambda}(x)\coloneqq\{y\in\mathcal{Y}_{\text{GT}}:S^{\downarrow}(x,y)\leq\lambda\}. (1) CP performs calibration by setting \hat{\lambda} to be the \tfrac{\lceil(n+1)\gamma\rceil}{n} quantile of S^{\downarrow}(X^{(1)},Y^{(1)}_{\text{GT}}),\dots,S^{\downarrow}(X^{(n)},Y^{(n)}_{\text{GT}}) . Then, \mathcal{C}_{\hat{\lambda}}(X^{(n+1)}) satisfies the CP desiderata: Theorem 1 ( Vovk et al. 1999 ) . Assume (X^{(1)},Y^{(1)}_{\textnormal{GT}}),\dots,(X^{(n+1)},Y^{(n+1)}_{\textnormal{GT}}) are exchangeable. Then, for \mathcal{C}_{\lambda} and \hat{\lambda} as defined above: {\mathbb{P}}\left(Y^{(n+1)}_{\textnormal{GT}}\in\mathcal{C}_{\hat{\lambda}}\left(X^{(n+1)}\right)\right)\geq\gamma. (2) Recall that the random variables Z^{(1)},\dots,Z^{(n+1)} are exchangeable if their joint distribution is invariant under any permutation of their indices. Exchangeability is a remarkably flexible assumption, as it is strictly weaker than the i.i.d. requirement. Two remarks are worth making about Theorem 1 . First, note that the probability in Equation 2 is taken with respect to all random quantities inside it, including the calibration dataset itself (upon which \hat{\lambda} depends). Second, although the conformal guarantee in Equation 2 holds regardless of the underlying data-generating distribution and the choice of S^{\downarrow} , the practical usefulness of CP does depend on S^{\downarrow} ; for example, a poor choice of S^{\downarrow} could lead to \mathcal{C}_{\hat{\lambda}}(X^{(n+1)})=\mathcal{Y}_{\text{GT}} , which would render the conformal guarantee vacuously true. Conformal Risk Control (CRC). Angelopoulos et al. 2024b extended CP to a more general setting where rather than controlling for coverage, the expectation of an arbitrary utility function is controlled instead. 1 1 1 CRC is usually formulated as upper-bounding a loss function. Lower-bounding a utility function is equivalent, and will facilitate comparisons to our method. Let \mathcal{C}_{\lambda}:\mathcal{X}\rightarrow 2^{\mathcal{Y}_{\text{GT}}} be a family of functions, indexed by \lambda\in\Lambda\coloneqq[\lambda_{\text{min}},\lambda_{\text{max}}] , producing prediction sets, and let U:2^{\mathcal{Y}_{\text{GT}}}\times\mathcal{Y}_{\text{GT}}\rightarrow[0,\infty] be a utility function on prediction sets. Intuitively, larger values of \lambda should