Blind Augmentation: Calibration-free Camera Distortion Model Estimation for Real-time Mixed-reality Consistency

\authororcid Siddhant Prakash0009-0000-8686-2442, \authororcid David R. Walton0000-0001-5879-9714, \authororcid Rafael K. dos Anjos0000-0002-2616-7541, \authororcid Anthony Steed0000-0001-9034-3020, and \authororcid Tobias Ritschel0009-0006-4660-7790 · 2025

Read Blind Augmentation: Calibration-free Camera Distortion Model Estimation for Real-time Mixed-reality Consistency by \authororcid Siddhant Prakash0009-0000-8686-2442, \authororcid David R. Walton0000-0001-5879-9714, \authororcid Rafael K. dos Anjos0000-0002-2616-7541, \authororcid Anthony Steed0000-0001-9034-3020, and \authororcid Tobias Ritschel0009-0006-4660-7790 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01387.txt.

Introduction

Siddhant Prakash and co-authors estimate camera blur on the fly so virtual objects blend into live video, in this 2025 paper. Real footage carries noise, motion blur, and soft focus zones. In augmented reality, virtual items must share those flaws or they look pasted on.

Older methods need a calibration step with a known target first. This blind method skips calibration and reads the distortion from the scene itself. It runs in real time, which mixed-reality headsets demand. Matching blur sells the illusion that digital and real share one camera. Games, design previews, and training tools all lean on that illusion. The team frame their estimator for live consumer hardware.

Catalog record

CREATOR\authororcid Siddhant Prakash0009-0000-8686-2442, \authororcid David R. Walton0000-0001-5879-9714, \authororcid Rafael K. dos Anjos0000-0002-2616-7541, \authororcid Anthony Steed0000-0001-9034-3020, and \authororcid Tobias Ritschel0009-0006-4660-7790
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01387

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Blind Augmentation: Calibration-free Camera Distortion Model Estimation for Real-time Mixed-reality Consistency \authororcid Siddhant Prakash0009-0000-8686-2442, \authororcid David R. Walton0000-0001-5879-9714, \authororcid Rafael K. dos Anjos0000-0002-2616-7541, \authororcid Anthony Steed0000-0001-9034-3020, and \authororcid Tobias Ritschel0009-0006-4660-7790 Abstract Real camera footage is subject to noise, motion blur (MB) and depth of field (DoF). In some applications these might be considered distortions to be removed, but in others it is important to model them because it would be ineffective, or interfere with an aesthetic choice, to simply remove them. In augmented reality applications where virtual content is composed into a live video feed, we can model noise, MB and DoF to make the virtual content visually consistent with the video. Existing methods for this typically suffer two main limitations. First, they require a camera calibration step to relate a known calibration target to the specific cameras response. Second, existing work require methods that can be (differentiably) tuned to the calibration, such as slow and specialized neural networks. We propose a method which estimates parameters for noise, MB and DoF instantly, which allows using off-the-shelf real-time simulation methods from e.g., a game engine in compositing augmented content. Our main idea is to unlock both features by showing how to use modern computer vision methods that can remove noise, MB and DoF from the video stream, essentially providing self-calibration. This allows to auto-tune any black-box real-time noise+ MB + DoF method to deliver fast and high-fidelity augmentation consistency. \onlineid 1842 \vgtccategory Research \vgtcpapertype system \preprinttext To appear in IEEE Transactions on Visualization and Computer Graphics. \ieeedoi xx.xxxx/TVCG.201x.xxxxxxx MB motion blur MR mixed reality DoF depth of field MLE maximum-likelihood estimation PSF point spread function std standard deviation \authorfooter Siddhant Prakash, Anthony Steed, and Tobias Ritschel are with University College London. E-mail: {s.prakash | a.steed | t.ritschel}@ucl.ac.uk . David R. Walton is with Birmingham City University. E-mail: [email protected] . Rafael K. dos Anjos is with University of Leeds. E-mail: [email protected] . \teaser Our method “blindly” estimates a model of noise, MB ( MB ) and DoF ( DoF ) from input frames (left), i.e., without requiring any known calibration markers / objects. The model can then augment other images with virtual objects that appear visually consistent (middle). Introduction keywords Augmented Reality, optimization. Video passthrough MR ( MR ) involves combining captured images and computer-generated images. Captured images come with many artifacts of the capture process such as motion blur (MB), depth of field (DoF), vignetting, noise, etc. While in recent mixed-reality headsets, capture is often set up so that artifacts are minimized, in many situations, they cannot be removed. Indeed captured images have characteristic aesthetic qualities that we might want to reproduce as is evident from the fact that many modern game engines provide methods to artificially introduce such artifacts in production footage for VFX. Even where possible, removing all such artifacts may be undesirable, and limit creative expression. Thus our goal is to make the computer-generated images visually consistent with the captured images by reproducing some of these artifacts. Real-time MR presents specific challenges within this larger problem. Convincing MR systems model several characteristics [ 3 ] : scene geometry, materials, illumination and parameters characteristics of the capture camera. MR systems run at high framerates, might use multiple cameras that have a relatively small form factor, and those cameras are attached to moving people or devices. Thus when processing video for MR systems, the system must deal with images with large amounts of noise, MB and DoF . Typically the solution is to calibrate the virtual camera, manually [ 13 , 9 , 1 , 24 ] or by means of clean-distorted image pairs [ 23 ] eventually used as training data for learning [ 21 ] . Each of these methods requires time and/or expert knowledge, requires markers and/or limits camera settings. Moreover, commercial video see-through augmented reality on mobile phones or head-mounted displays (HMD) may apply closed-box post-processing algorithms to the images, using unknown parameters. An ideal MR system would not require calibration and could deal with dynamic changes in focus, noise, lighting levels, etc. We propose blind augmentation (Fig. Blind Augmentation: Calibration-free Camera Distortion Model Estimation for Real-time Mixed-reality Consistency ), which can achieve visual consistency between captured and synthesized images, but without active calibration. It is “blind” in the sense of “blind deconvolution” [ 16 , 17 ] , which can reconstruct a distortion (e.g., a convolution, i.e., a blur) without knowing the precise nature of the particular distortion. The twist in our work is that we are not seeking to remove distortion from real videos, but to blindly introduce distortion to virtual content to make it consistent with the captured images. The key to make this work is recent methods to extract depth and motion as well as to remove noise, MB and DoF from video streams. Our contributions are: • A method to estimate parameters for noise, MB and DoF from a wild video without the need of calibration patterns, or camera metadata/hardware information. • Parameter update with an upper bound of 1.5\text{\,}\mathrm{s} ec., which allows video see-through augmented reality to be composited with consistent quality shortly after start-up. • A demo application demonstrating effectiveness of our method on the Meta Quest 3 HMD running at 45\text{\,}\mathrm{Hz} . All code and datasets used in the paper can be found on the project page: https://prakashsidd18.github.io/projects/blind_augmentation/ . 1 Previous work Our work tackles visual consistency between live video and an augmented virtual rendering as required by real-time MR systems [ 4 , 32 , 6 ] . Certainly, the view pose, geometry, reflectance and light need to be modeled or estimated, as pioneered by Fournier et al. [ 10 ] and reviewed by Alhakamy and Tuceryan [ 2 ] . In our work we assume these are known. Instead, we look into the specific problem of recreating distortions that are found in a live video stream. Klein and Murray [ 14 ] , Fischer et al. [ 9 ] , Aittala [ 1 ] and Park et al. [ 24 ] simulate a number of image degradation effects for a particular small wide-angle camera, including barrel distortion, color response, chromatic aberration and vignetting. Motion blur due to camera rotation (but not translation) is also simulated. An early work in this area, Okumura et al. [ 23 ] , fits a PSF ( PSF ) to a real image based on the appearance of a circular AR marker placed in the image. This is then convolved with virtual content to add motion blur and DoF effects. Its main limitations are the need for a known AR marker to be present in the scene, and estimation of a single global PSF . This means the approach can only handle static virtual objects at a similar depth to the marker. We handle dynamic virtual objects at any depth, without the need for known AR markers in the scene. The most strongly related work to our own is Mandl et al. [ 21 ] . The authors estimate camera parameters to introduce depth of field blur, vignetting and sensor artifacts such as noise. They fit a complex camera model through capturing calibration images of a priori known markers and color

Read the complete text