Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMs
Hen Davidov, Gilad Freidkin, Shai Feldman, Yaniv Romano · 2025
Read Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMs by Hen Davidov, Gilad Freidkin, Shai Feldman, Yaniv Romano (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13593.txt.
Introduction
Hen Davidov and co-authors define a new safety measure for language models in a 2025 paper. The measure counts how many outputs a model must produce before one turns unsafe. Unsafe outputs are rare in well-tuned models, so the count is hard to pin down. A test budget may end before any bad output appears.
The authors treat the count as a survival problem, like time until an event. They add a calibration method that gives a lower bound for the count, with a stated level of trust. A smart plan for spending the test budget improves efficiency. Tests on made-up and real data backed the theory. The measure lets testers judge safety prompt by prompt.
Catalog record
| CREATOR | Hen Davidov, Gilad Freidkin, Shai Feldman, Yaniv Romano |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13593
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMs Hen Davidov, Gilad Freidkin, Shai Feldman, Yaniv Romano Abstract We develop a framework to quantify the time-to-unsafe-sampling—the number of large language model (LLM) generations required to trigger an unsafe (e.g., toxic) response. Estimating this quantity is challenging, since unsafe responses are exceedingly rare in well-aligned LLMs, potentially occurring only once in thousands of generations. As a result, directly estimating time-to-unsafe-sampling would require collecting training data with a prohibitively large number of generations per prompt. However, with realistic sampling budgets, we often cannot generate enough responses to observe an unsafe outcome for every prompt, leaving the time-to-unsafe-sampling unobserved in many cases, making the estimation and evaluation tasks particularly challenging. To address this, we frame this estimation problem as one of survival analysis and develop a provably calibrated lower predictive bound (LPB) on the time-to-unsafe-sampling of a given prompt, leveraging recent advances in conformal prediction. Our key innovation is designing an adaptive, per-prompt sampling strategy, formulated as a convex optimization problem. The objective function guiding this optimized sampling allocation is designed to reduce the variance of the estimators used to construct the LPB, leading to improved statistical efficiency over naive methods that use a fixed sampling budget per prompt. Experiments on both synthetic and real data support our theoretical results and demonstrate the practical utility of our method for safety risk assessment in generative AI models. 1 Introduction Recent advances in large language and vision models have unlocked exciting possibilities, but also introduced new safety challenges. To illustrate, suppose we are given a large language model (LLM) \mathcal{G} , which gets the prompt X as an input, and produces a response \mathcal{G}(X) . For example, consider the prompt “Can you tell me an old-school joke that wouldn’t be considered appropriate today.” Although LLMs are trained to be aligned with human preferences— e.g., [ 43 , 59 , 6 , 20 ] —they may still produce unsafe outputs, even for non-adversarial inputs with some probability [ 4 , 55 , 47 ] . In this paper, we focus on a more common setting rather than an adversarial one (e.g., that use jailbreaking), where resampling an LLM many times—or having many users sample it in parallel—could eventually produce an unsafe output. In our illustrative example, this could be a “joke” that marginalizes minorities. We define unsafe output through an external classifier or human rubric \texttt{Audit}(X,\mathcal{G}(X))\in\{0,1\} , which flags, for example, toxic language that undermines user trust [ 55 , 11 ] , outputting private, sensitive information [ 25 , 13 ] , or deepfakes that defame individuals [ 63 ] , to name a few. One possible solution to protect users from unsafe generations is to audit the outcome at inference time using some Audit function [ 21 , 65 , 19 ] . This allows, for example, to generate more responses until one of the responses is considered safe [ 1 , 3 , 53 , 10 , 64 , 32 ] . In this paper, we complement the retrospective safety auditing of generated responses with a proactive approach. Specifically, for a given prompt X and generative model \mathcal{G} , we ask: “How many safe responses can we expect before encountering an unsafe one?” Ideally, this foresight-driven analysis offers early insights into the risk level of a prompt. For instance, if the expected number of safe generations is low for a particular prompt, it signals the need for more sophisticated—and thus more resource-intensive—safety checks. This approach also enables assessing a model’s reliability on critical queries and comparing different models based on their predicted risk on a set of prompts. A direct strategy for estimating this risk is to train a regression model that, given a prompt, predicts the number of safe generations before the first unsafe one appears—what we refer to as the time-to-unsafe-sampling . Naturally, to fit such a model, we should annotate the data. Ideally, this can be done by generating multiple outcomes for each prompt until an unsafe outcome is observed. Unfortunately, with well-aligned LLMs, the expected number of safe generations can be astronomically large; for example, a prompt might require samples on the order of 10^{8} before having a substantial chance of producing a violation. Taking a more practical approach, we assume a budget constraint B that limits the total number of LLM generations and their audits. This constraint reflects realistic finite computational resources and human review capacity. As a consequence of the limited budget, most real-world datasets will have many prompts without a fully observed time-to-unsafe-sampling. In fact, under such a sampling budget constraint, we might never observe time-to-unsafe-sampling labels, even for prompts that have a moderate unsafe output probability. This highlights a fundamental limitation of a naive use of a regression model estimating the time-to-unsafe-sampling outcome for a given prompt. The model forecast may be systematically uncalibrated, giving over- or under-estimates for time-to-unsafe-sampling with no built-in reliability guarantee. Fortunately, we can construct a reliable time-to-unsafe-sampling estimate by considering this LLM prompt risk assessment as a survival analysis task. In survival analysis, the goal is to predict a patient’s true survival time T —e.g., time to mortality—from covariates X . However, the target variable T is not fully observed for all patients due to censoring time C ; e.g., end of the clinical trial or a patient’s withdrawal of consent. Hence, the survival data are formulated as triplets of (X,C,\tilde{T}) , each with a censored survival time \tilde{T}=\min(T,C) (see [ 39 , 24 ] ). In this work, we treat the prompt risk assessment task as a survival analysis problem, allowing us to build on the foundations of a well-studied field. This mapping is done by considering the survival time T as the time-to-unsafe-sampling output and the censoring time C as the maximum number of generation-and-audit cycles allocated to estimate the risk of the prompt X . Building on recent conformalized survival analysis techniques [ 15 , 5 , 9 , 50 ] , our method proceeds in two stages. First, we train a regression model on a subset of prompts to predict the time-to-unsafe-sampling. Next, we use a holdout set of prompts to calibrate the prediction of the regression model and obtain a lower predictive bound (LPB), denoted by \hat{L}(X) , on the time-to-unsafe-sampling. In practice, this calibrated LPB estimates the time-to-unsafe-sampling while controlling the miscoverage rate—the rate at which the actual (unknown) time-to-unsafe-sampling is lower than the LPB at test time. Informally, this guarantee allows us to state that, with high probability, one can generate at least \hat{L}(X) safe responses before encountering an unsafe one, on average over future test prompts. Crucially, computing \hat{L}(X) at inference time requires only a single call to the calibrated regression model and does not involve running the Audit model or sampling multiple LLM generations. Our contributions A key difference from previous conformalized survival analysis methods is that we can explicitly design the per-prompt censoring times C under a global sampling budget B . We show that naively splitting the budget equally across all samples can result in LPBs with high variability in their actual miscoverage rate. To address this, we introduce prompt adaptive allocation schemes for determining C , which substantially reduce