Ultra-low-light computer vision using trained photon correlations

Mandar M. Sohoni, Jérémie Laydevant, Mathieu Ouellet, Shi-Yuan Ma, Ryotatsu Yanagimoto, Benjamin A. Ash, Tatsuhiro Onodera, Tianyu Wang, Logan G. Wright, Peter L. McMahon · 2026

Read Ultra-low-light computer vision using trained photon correlations by Mandar M. Sohoni, Jérémie Laydevant, Mathieu Ouellet, Shi-Yuan Ma, Ryotatsu Yanagimoto, Benjamin A. Ash, Tatsuhiro Onodera, Tianyu Wang, Logan G. Wright, Peter L. McMahon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11993.txt.

Introduction

Sohoni and nine colleagues use correlated-photon illumination to boost object recognition in ultra-low-light, noisy imaging. Signal photons match up in space while noise clicks do not. Earlier work used that fact to rebuild images; here the goal is inference instead. Their CAT method jointly tunes a trainable photon source and a Transformer backend, using at most 100 shots. Accuracy rises by up to 15 points over standard uncorrelated light, and beats untrained correlated light too.

Catalog record

CREATORMandar M. Sohoni, Jérémie Laydevant, Mathieu Ouellet, Shi-Yuan Ma, Ryotatsu Yanagimoto, Benjamin A. Ash, Tatsuhiro Onodera, Tianyu Wang, Logan G. Wright, Peter L. McMahon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE108359
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.11993

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ultra-low-light computer vision using trained photon correlations Mandar M. Sohoni, Jérémie Laydevant, Mathieu Ouellet, Shi-Yuan Ma, Ryotatsu Yanagimoto, Benjamin A. Ash, Tatsuhiro Onodera, Tianyu Wang, Logan G. Wright, Peter L. McMahon Abstract Illumination using correlated photon sources has been established as an approach to allowing high-fidelity images to be reconstructed from noisy camera frames by taking advantage of the knowledge that signal photons are spatially correlated whereas detector clicks due to noise are uncorrelated. However, in computer-vision tasks, the goal is often not ultimately to reconstruct an image, but to make inferences about a scene—such as what object is present. Here we show how correlated-photon illumination can be used to gain an advantage in a hybrid optical–electronic computer-vision pipeline for object recognition. We demonstrate correlation-aware training (CAT): end-to-end optimization of a trainable correlated-photon illumination source and a Transformer backend in a way that the Transformer can learn to benefit from the correlations, using a small number ( ≤ \leq 100) of shots. We show a classification accuracy enhancement of up to 15 percentage points over conventional, uncorrelated-illumination-based computer vision in ultra-low-light and noisy imaging conditions, as well as an improvement over using untrained correlated-photon illumination. Our work illustrates how specializing to a computer-vision task—object recognition—and training the pattern of photon correlations in conjunction with a digital backend allows us to push the limits of accuracy in highly photon-budget-constrained scenarios beyond existing methods focused on image reconstruction. I Introduction Optical imaging has widely been used to probe and then control the dynamics of physical systems. In cases where the goal is to make inferences about entities in an image, for example classifying cell organelles in images of cells, the extraction of task-specific information from high-dimensional images is increasingly relying on computer vision. A typical computer vision pipeline can be thought of as follows. A camera provides a high-dimensional digital image of a scene and a digital-neural-network extracts information from the image to make inferences about the scene. To enhance the efficacy of information extraction from high-dimensional image data, the design of the image-acquisition hardware (the camera and optics in front of it) has become inextricably linked with the image-processing software through methods such as end-to-end optimization Wetzstein et al. 2020 . Conceptually, all of these methods operate as an encoder-decoder architecture: a physical-domain encoder extracts scene information and converts it into measurable signals, leaving a digital-domain decoder to interpret the data for specific inference tasks. Concurrently, the field of photonic quantum sensing has explored the fundamental limits of information carried by photons, developing advanced illumination and measurement schemes to approach these limits Pirandola et al. 2018 ; Magana-Loaiza and Boyd 2019 ; Moreau et al. 2019 ; Defienne et al. 2024 ; Tsao et al. 2025 . Many of these enhancements arise from leveraging photon correlations in the state of light. For instance, squeezed states have improved microscope performance Casacio et al. 2021 , and N 00 N states have benefited interferometers Nagata et al. 2007 ; Israel et al. 2014 ; Slussarenko et al. 2017 , building on the initial proposal that such correlations could mitigate the effects of shot noise Caves 1981 . Light sources with intrinsic spatial correlations between photons, such as those produced by spontaneous parametric down-conversion (SPDC), have enabled the development of novel imaging techniques. These techniques have achieved sub-shot-noise imaging Brida et al. 2010 ; Taylor et al. 2013 ; Zhang et al. 2024 , background-light suppression Lloyd 2008 ; Dylov et al. 2011 ; Lopaeva et al. 2013 ; Gregory et al. 2020 , ghost imaging Pittman et al. 1995 ; Strekalov et al. 1995 ; Bennink et al. 2002 , pattern recognition Ortolano et al. 2023 , and enhanced resolution Giovannetti et al. 2009 ; Toninelli et al. 2019 ; He et al. 2023 . By accounting for the illumination’s intrinsic photon statistics, these methods capitalize on a perspective often overlooked by their uncorrelated counterparts: higher-order photon correlations can carry information distinct from the mean field 1 1 1 Here mean field refers to the average intensity of the illumination pattern, i.e., a vector consisting of the mean photon number for each camera pixel. . This information is particularly valuable in low-light settings, as these correlations demonstrate a natural robustness Grochowski et al. 2026 to uncorrelated noise such as that from background light or camera readout. While these techniques have shown a lot of promise, harnessing the information embedded within these higher-order correlations presents a practical challenge. Figure 1 : Ultra-low-light computer vision with uncorrelated and correlated illumination. a, A comparison of conventional and our proposed computer vision pipelines. (i) , A conventional computer vision pipeline. A light source emits uncorrelated photons towards an object and the photons that are not blocked by the object are recorded on a single-photon-sensitive camera. The shots (frames) from the camera are sent to a Set Transformer backend to classify which object was being imaged. (ii) , A computer vision pipeline with a correlated-photon illumination source. Instead of uncorrelated photons as in (i), the light source emits spatially correlated photons. (iii) , Our proposed computer vision pipeline with a trainable-correlated-photon source. The light source emits trainable correlated photons that, when trained, allow the object to classified with fewer shots and/or photons. Appendix A introduces an example of a classification task where photon correlations are useful. b, Correlation-aware training (CAT). A schematic of the end-to-end optimization loop used to train photon correlations in a computer vision pipeline. A Set Transformer uses shots from the camera to classify the object being image, and backpropagates the error through a digital model of the correlated light source to update the photon correlations and the weights of the Set Transformer. Estimating the photon correlations or a projection of these correlations into a lower-dimensional space with high fidelity typically requires averaging over a prohibitive number of shots Defienne et al. 2024 . This necessity for extensive data acquisition can hinder the advantages of correlated illumination, particularly in applications where a low-photon budget or low-latency operation is an inherent constraint. State-of-the-art digital image classification has been revolutionized by neural networks designed to detect statistical correlations within image pixels. Vision Transformers, for example, use self-attention mechanisms to learn long-range relationships, such as determining that a patch of pixels representing a “dog’s ear” is statistically related to another patch representing a “dog’s tail”. However, the efficacy of these networks is fundamentally limited by the quality of the input data; in noisy conditions, such as when one is operating with ultra-low-light illumination, the statistical correlations these algorithms rely on can become obscured and difficult to detect without acquiring a large number of shots. We hypothesize that merging correlated illumination techniques with an advanced Transformer backend can enhance ultra-low-light computer vision. This expectation is twofold: (1), the Transformer’s self-attention mechanisms—conceptually linked to Gram and covariance matrices El-Nouby et al. 2021 —are theoretically well-suited to efficiently extract information from photon correlations; and (2), the deterministic relationship between photon pairs should provide a uniquely robust signal for the network to lock onto, surpassing purely statistical correlations. Figures 1 a (i) and (ii) outline this concept by contrasting a computer vision pipeline using an uncorrelated-photon source with one using a correlated-photon

Read the complete text