When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation
Varun Kotte · 2026
Read When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation by Varun Kotte (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29054.txt.
Description
Kotte asks when a statistical method called conformal risk control can certify the outputs of language models used for structured tasks like extracting names or filling JSON forms. Such models lack formal reliability guarantees, and common abstention rules miss user-set risk targets widely. He characterises both where certification works and where it provably cannot.
First comes an impossibility result. If a model's base risk exceeds the target, any distribution-free method must abstain on a minimum share of examples, which he computes in closed form. That gives a feasibility test to run before attempting certification. He then ranks three certification bounds. The middle one delivers the largest gain in certified configurations, while a betting-based bound helps when calibration data is scarce. Under dataset shift, an adaptive method cut risk-target violations from 71 to 21 percent. Hard tasks stayed uncertifiable at strict risk levels but opened up at looser ones.
Catalog record
| CREATOR | Varun Kotte |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29054