LRT-Diffusion: Calibrated Risk-Aware Guidance for Diffusion Policies
Ximan Sun, Xiang Cheng · 2025
Read LRT-Diffusion: Calibrated Risk-Aware Guidance for Diffusion Policies by Ximan Sun, Xiang Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24983.txt.
Introduction
Ximan Sun and Xiang Cheng add a risk aware rule to spread based plans for offline trial based learning. Such plans are strong, but their run time guide is most often a fixed push with no sense of risk. Their rule treats each clean up step as a test between two takes on the next move.
A log score builds as steps go, and a gate with a set line lets the state based mean through in part. The line is set once to meet a user picked false call rate, so the risk cap can be read in plain terms. Training stays plain; the new part is just at run time. Q slope pushes can join at more than one point.
Catalog record
| CREATOR | Ximan Sun, Xiang Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LRT-Diffusion: Calibrated Risk-Aware Guidance for Diffusion Policies Ximan Sun, Xiang Cheng Abstract Diffusion policies are competitive for offline RL but are typically guided at sampling time by heuristics that lack a statistical notion of risk. We introduce LRT - Diffusion, a risk - aware sampling rule that treats each denoising step as a sequential hypothesis test between the unconditional prior and the state - conditional policy head. Concretely, we accumulate a log - likelihood ratio and gate the conditional mean with a logistic controller whose threshold τ \tau is calibrated once under H 0 H_{0} to meet a user - specified Type - I level α \alpha . This turns guidance from a fixed push into an evidence-driven adjustment with a user-interpretable risk budget. Importantly, we deliberately leave training vanilla (two heads with standard ϵ \epsilon -prediction) under the structure of DDPM. LRT guidance composes naturally with Q - gradients: critic - gradient updates can be taken at the unconditional mean, at the LRT - gated mean, or a blend, exposing a continuum from exploitation to conservatism. We standardize states/actions consistently at train and test time and report a state - conditional OOD metric alongside return. On D4RL MuJoCo tasks, LRT - Diffusion improves the return–OOD trade - off over strong Q - guided baselines in our implementation while honoring the desired α \alpha . Theoretically, we establish level - α \alpha calibration, concise stability bounds, and a return comparison showing when LRT surpasses Q - guidance—especially when off - support errors dominate. Overall, LRT - Diffusion is a drop - in, inference - time method that adds principled, calibrated risk control to diffusion policies for offline RL. K eywords Offline Reinforcement learning; Diffusion Policies; Likelihood - ratio Test; Inference; Risk - aware Guidance; Calibration; Distribution Shift; Q - guidance; Out - of - distribution Detection; D4RL. 1 Introduction Offline reinforcement learning (RL) aims to learn high - performing policies from fixed datasets without further environment interaction. A central difficulty is distributional shift : actions proposed by the learned policy can drift away from the behavioral support where value estimates are reliable. Diffusion policies have recently emerged as strong generative decision - makers for offline RL Janner2022Diffuser : by learning a conditional diffusion model over actions given state, they produce smooth, high - fidelity samples that respect support better than direct regression. However, how these policies are guided at sampling time remains largely heuristic. Common practices—injecting Q - gradients with hand - tuned schedules and ad - hoc clipping—lack a statistical notion of risk and offer limited control of the return–shift trade - off. Unlike prior works, our approach keeps training strictly vanilla (no critic-guided losses) and moves all risk control to inference via a calibrated likelihood-ratio gate (LRT), yielding an interpretable, reproducible risk knob. Intuition. Rather than always pulling samples toward the conditional head, we ask at every denoising step: is there enough evidence to move toward the “good - action” direction for this state? Concretely, we split offline actions into a good subset and a background subset, train a two - head diffusion model (an unconditional head on all data and a conditional head on good data) with class - balancing and optional advantage - based soft weights, and then gate the conditional pull at inference by a calibrated likelihood ratio. The gate is motivated by the Neyman–Pearson test NeymanPearson1933 : a single user knob Type - I rate \alpha controls the tolerated false activations under H_{0} (“background is correct”); while the hard LRT is UMP at level \alpha under equal covariances, we use a smooth gate in practice for numerical stability and keep \alpha interpretable via calibration on held - out states. Method overview. We introduce LRT - Diffusion , a risk - aware, inference - only sampling scheme for diffusion policies. At each denoising step we make a binary decision between a background prediction and a data - conditioned prediction. We accumulate evidence during sampling and open the gate only when the evidence is strong, interpolating between the two predictions with a data - dependent weight. A single threshold is calibrated once so that the empirical false - activation rate does not exceed a user - chosen \alpha , turning guidance from a fixed push into an evidence - driven adjustment: weak evidence \Rightarrow stay near the background prior; strong evidence \Rightarrow move decisively toward the conditional policy. Training is unchanged. Composition with value guidance. We optionally combine LRT gating with a small critic step: at each denoising update, take a capped step that increases the learned critic at the current proposal (standard decreasing schedule). The gradient can be evaluated near the background proposal, the LRT-gated proposal, or an interpolation; the threshold is calibrated with the same choice, so the Type-I semantics are preserved. Theoretical guarantees at a glance. Under equal - covariance heads, the hard gate(thresholding under LRT assumptions) is the uniformly most powerful test at level \alpha ; our soft gate (a numerically stable surrogate) retains the threshold’s semantics and improves stability. Beyond testing optimality, we analyze offline - RL error propagation and show that reducing the policy’s state - conditional OOD rate—which LRT lowers by construction—tightens a lower bound on true return relative to pure Q - guidance when the critic is unreliable off - support. Practical picture. The gate exposes monotone, interpretable knobs: \alpha (risk), \beta_{\max} (max pull), and \delta (sharpness). LRT can be combined with a small critic step evaluated at the background mean, the LRT-gated mean, or a blend; the threshold is calibrated with that same choice so the Type-I semantics are preserved. Empirically on D4RL MuJoCo, LRT tracks the target \alpha and improves the return–OOD Pareto relative to strong Q -guided baselines in our implementation. 1.1 Contributions Calibrated LRT guidance for diffusion policies. Unlike prior research integrating value or energy guidance into the diffusion model, we cast each denoising step as a likelihood-ratio test between a background head and a “good” head, and calibrate a single threshold \tau on held-out states to bound the trajectory-level Type-I rate at a user-chosen \alpha . Training remains vanilla (two heads with class balancing and optional advantage weights). Labeling-and-weighting recipe for the conditional head. Rather than fitting a single conditional diffusion model to all (state, action) pairs, as is common in return-dominated diffusion or Q -guided training, we split offline actions into a good subset and a background subset using IQL advantages (top- p quantile), and train a two-head diffusion model with class-balancing and advantage-aware soft weights. This improves the quality of the conditional head without extra supervision. 1 1 1 E.g., Diffuser/Decision - Diffuser use return conditioning and sampling-time guidance; Diffusion - QL injects value signals into the training loss, all without a binary good/background head. Actionable theory with finite-sample and stability guarantees. Under equal covariances the hard LRT is UMP at level \alpha (Prop. 1 ); a one-shot Monte-Carlo calibration yields \mathbb{P}_{H_{0}}(\ell_{\mathrm{cum}}\geq\hat{\tau})\leq\alpha+\varepsilon_{n} (Thm. 2 ). We bound the per-step deterministic drift and show the cumulative LLR is sub-Gaussian with an explicit variance proxy (Sec. 5.2 ), explaining why calibration is numerically stable. Return bounds that connect risk control to distribution shift. With a standard offline error split, we prove a comparison