Local Conformal Calibration of Dynamics Uncertainty from Semantic Images

Luís Marques, Dmitry Berenson · 2026

Read Local Conformal Calibration of Dynamics Uncertainty from Semantic Images by Luís Marques, Dmitry Berenson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13028.txt.

Introduction

Marques and Berenson introduce OCULAR, a conformal prediction method that calibrates dynamics uncertainty from semantic images. Older methods cannot tell which state-action regions carry higher model mismatch and need data tied to one setting. OCULAR instead uses data from look-alike settings to calibrate a linear Gaussian dynamics model, with prediction regions guaranteed to hold future states at a user-set likelihood. Tests on a noisy double-integrator system with big model mismatch confirm it works in and out of distribution.

Catalog record

CREATORLuís Marques, Dmitry Berenson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71970
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13028

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Local Conformal Calibration of Dynamics Uncertainty from Semantic Images Luís Marques, Dmitry Berenson Abstract We introduce O bservation-aware C onformal U ncertainty L ocal-C a lib r ation ( OCULAR ), a conformal prediction-based algorithm that uses perception information to provide uncertainty quantification guarantees for unseen test-time environments. While previous conformal approaches lack the ability to discriminate between state-action space regions leading to higher or lower model mismatch, and require environment-specific data, our method uses data collected from visually similar environments to provably calibrate a given linear Gaussian dynamics model of arbitrary fidelity. The prediction regions generated from OCULAR are guaranteed to contain the future system states with, at least, a user-set likelihood, despite both aleatoric and epistemic uncertainty—i.e., uncertainty arising from both stochastic disturbances and lack of data. Our guarantees are non-asymptotic and distribution-free , not requiring strong assumptions about the unknown real system dynamics. Our calibration procedure enables distinguishing between observation-velocity-action inputs leading to higher and lower next-state-uncertainty, which is helpful for probabilistically-safe planning. We numerically validate our algorithm on a double-integrator system subject to random perturbations and significant model mismatch, using both a simplified sensor and a more realistic simulated camera. Our approach appropriately quantifies uncertainty both when in-distribution and out-of-distribution, being comparatively volume-efficient to baselines requiring environment-specific data. Project website: https://um-arm-lab.github.io/ocular 1 Introduction Provable uncertainty quantification typically requires imposing restrictive assumptions on the distribution of external disturbances [ 6 ] or assuming a lack of model mismatch [ 1 ] . However, real systems are often subject to disturbances that are difficult to model analytically (e.g., near-wall aerodynamic effects) and are action-dependent (e.g., tire slip on ice can vary with velocity). The discrepancy between the strict conditions required for classical guarantees and realistic deployment conditions can lead to unsafe actions. Conversely, while learned dynamical systems have made progress towards modeling complex dynamics [ 24 ] , their predictions are unreliable when operating outside the training regime. It remains challenging to provide theory-grounded guarantees on the uncertainty present in next-state dynamics prediction when subject to unknown disturbances and when relying on a dynamics model of unknown fidelity. Conformal Prediction (CP) has recently emerged as a statistical framework for distribution-free uncertainty quantification. CP does not limit the form of the unknown dynamics or the external disturbances. CP provides provable probabilistic guarantees given finite-sized amount of calibration data , enabling a data-driven but theoretically-grounded quantification of prediction uncertainty. However, CP still lacks some properties limiting its applicability in robotics. The CP guarantees are conventionally given on average over a set of possible conditions, and require system-transition data representative of test-conditions. Hence, calibration data is conventionally collected from the same environment as test-time deployment [ 17 , 19 , 20 ] —a significant shortcoming. Besides poor generalizability, per-environment data-collection can expose the system to unnecessary risks. For example, if the CP guarantees depend on robot poses, then we might have to “completely” traverse a new road to determine in which of its segments our dynamics model is more or less accurate (e.g., which parts are icy). However, if the guarantees depend instead on perception information (e.g., a semantic image of the pavement ahead of the vehicle), then we might be able to quantify uncertainty in scenarios that have not yet been tested, but that are visually-similar to previous examples. That is, we could possibly learn that going fast on icy roads might lead to high dynamics uncertainty, without having to test all icy roads independently. In this paper we tackle this limitation, by introducing a CP-based algorithm that utilizes robot-frame perception information to achieve the same coverage guarantees as existing approaches without requiring any calibration data from the test environment . Instead, we utilize data from different, but perceptually-similar environments, and a learned representation of observations to safely generalize to unseen situations. Further, we account for how uncertainty might vary across velocity-action-observation contexts, and provide an adaptive calibration of approximate Gaussian uncertainty estimates, enabling the construction of a prediction region that will contain the unknown next state with a user-set likelihood of (1-\upalpha)\in(0,1) . Our key contributions are i) We propose an algorithm that, given state, action, and perception information from visually-similar environments, provides provable finite-sample guarantees on dynamics uncertainty in a new unseen environment. ii) We prove the validity of our approach combining learned representations and local conformal calibration, and demonstrate how it can be deployed to construct safe plans that steer the system towards lower-uncertainty states. iii) We validate our approach on a double-integrator system using both planar-sensors and Isaac Sim cameras, demonstrating comparable safety and performance to methods requiring environment-specific data. 1 1 1 See https://um-arm-lab.github.io/ocular for planning videos of all methods. 2 Related Work Safe planning under uncertainty has a long history spanning approaches from tube-based MPC [ 27 ] , contigency-based MPC [ 2 ] , using Gaussian-process to quantify aleatoric uncertainty [ 13 ] , ensemble-based models to quantify epistemic uncertainty [ 30 ] , etc. Classical approaches, such as control barrier functions [ 3 ] or reachability-based planning [ 5 ] , may be computationally expensive online or require a certain system structure. Conversely, learning-based methods often provide looser uncertainty estimates, especially when out-of-distribution (OOD). In this work, we use a data-driven and theoretically-grounded approach to provide probabilistic guarantees on one-step safety, and a volume-efficient calibration of approximate Gaussian uncertainty models. Conformal Prediction in robotics. CP has been used for socially-aware robot navigation [ 17 , 10 ] , expert imitation [ 28 ] , failure detection [ 18 ] , synthesis of control barrier functions [ 33 ] , and calibration of Lie-algebraic uncertainty estimators [ 20 ] . However, existing methods perform a state-, action-, and observation- agnostic calibration of system uncertainty. Yet, for some robot system and approximate dynamics models, the uncertainty underlying a given state-transition may not be constant. Empirically, global calibration methods have limited utility for safe plan generation, as these cannot steer the system towards lower-uncertainty states. While [ 19 ] proposed a position-velocity-action-dependent calibration, its reliance on world-frame poses requires collecting data on each deployment environment. In contrast, we leverage perception information to enable local calibration of dynamics uncertainty in environments for which we have not collected any data . While Online CP methods [ 10 , 11 ] do not assume a calibration dataset, these provide input-independent asymptotic guarantees. Instead, our velocity-action-observation-dependent guarantees are valid for finitely-sized datasets collected in different environments than the test environment. Conformal Prediction with sensing. CP has been used as wrapper on vision-based state-estimators, to probabilistically bound unmodeled sensing uncertainty and lead to theory-grounded state estimates [ 32 , 26 ] . In [ 21 ] , bounding-box predictors and occupancy predictors were calibrated to build regions containing the true obstacles at a specified likelihood. While useful for provably safe-autonomy, these works perform an input-independent calibration of learned sensing modules. We do not focus on state or obstacle estimation, assuming the current robot pose and velocity to be known, and that obstacles can be adequately observed. Rather, we provide an observation-dependent local calibration of system transition uncertainty, which we demonstrate can lead to more efficient and safe plans than CP baselines performing global calibration. 3 Problem Statement Let s_{t}\in\mathcal{S} and

Read the complete text