Conditions for well-posed color recovery in scattering media Grigory Solomatov, Derya Akkaynak Hatter Department of Marine Technologies, University of Haifa, Haifa, IsraelInteruniversity Institute for Marine Sciences, Eilat, Israel Abstract Recovering scene color from images captured in scattering media is a fundamental inverse problem in optical imaging. Yet the problem is intrinsically ill-posed as multiple solutions can explain the same observation, and prediction error cannot be controlled without understanding the space of candidate solutions. Here, we present sufficient conditions under which color recovery in a scattering medium becomes well-posed. Observing that ill-posedness stems from (i) projection of spectral signals onto pixel intensities, and (ii) unknown medium parameters, we demonstrate that sensor improvements alone cannot resolve medium-induced distortions without additional constraints. We identify recovery patterns —cross-pixel relationships that naturally occur in images—and prove, for an ideal hyperspectral camera, that they restrict the solution to a unique candidate. This opens the door to a new class of vision algorithms grounded in first principles, enabling quantitative analysis of images in scattering environments. Image data, while ostensibly an assignment of color values to spatial coordinates, has a remarkable capacity to encode information about the captured scene. The pursuit of automating the extraction of this visual information created the discipline of computer vision, yielding robust and powerful solutions for classification, segmentation, and tracking [ 8 ] . While these advancements have driven substantial progress in fields ranging from autonomous navigation and industrial quality control to medical imaging and ecology, a significant performance gap remains for computer vision in scattering media [ 4 , 14 , 19 , 25 , 24 ] . In contrast to what happens to light as it travels through clear air, which is more or less nothing, a scattering medium transforms light exponentially with distance [ 17 ] . In photographs, this transformation has the effect of distorting colors, reducing contrast, and blurring textures on which standard computer vision algorithms are based [ 2 , 4 , 19 ] . Furthermore, the spectral radiance from the scene toward the camera (inherent scene radiance) cannot directly be linked to the pixel values in the image. This remains true even in RAW format, which linearly encodes the spectral radiance incident at the camera (apparent scene radiance) projected onto the camera’s spectral sensitivity functions [ 1 ] . This limitation neither arises from the non-linear internal processing of consumer cameras, nor from the spectral compression inherent to RGB imaging, but from the unknown modification imposed by the medium itself. As a result, estimating the inherent radiance of a scene in a scattering environment is a fundamentally ill-posed problem, even for a hyperspectral camera with perfect accuracy (see Preliminaries). Addressing this ill-posedness is essential for many computer vision applications [ 3 , 14 , 5 , 7 , 28 , 26 , 18 ] , but it remains unclear whether, and under what conditions, the inherent radiance of scene captured in a scattering medium is recoverable from image data. Without this theoretical foundation, progress is driven by empirical performance rather than by an understanding of what the data can determine in principle, as ill-posed inverse problems can produce visually plausible but fundamentally ambiguous solutions (Fig. 1 ). Fig. 1: Ill-posed problems can have ambiguous solutions . This conceptual example demonstrates that an observed ‘bluish’ image can arise from a colorful scene under a blue illuminant, or a genuinely ‘bluish’ scene under white light, among other possibilities. Without further information, it is impossible to find a unique solution. Here, we address this gap by asking: under what conditions does color recovery become well-posed? By separating ambiguities introduced by the camera from those imposed by the unknown medium, we identify sufficient conditions for eliminating the latter. This perspective shifts the focus from improving algorithms within an ill-posed formulation to reformulating the problem itself so that recovery is provably well-posed. These challenges are perhaps most pronounced in an underwater imaging setting, where the collection of imagery is rapidly expanding through robotics, climate monitoring, and ecological surveys, yet its scientific use remains fundamentally limited by color distortions introduced by water and its constituents [ 16 ] . Correcting these distortions has been treated primarily as an engineering challenge, and most works in the last three decades have presented incremental algorithmic advances. Hence, we use this setting as a representative example, but our analysis and results are broadly applicable to all scattering conditions in which the same image formation model holds (described in the Preliminaries). Color recovery Fig. 2: Color recovery in a scattering medium is inherently ill-posed. Here, given an unprocessed RAW image (left) taken underwater, scene colors are recovered with the Sea-thru algorithm [ 3 ] (right), currently the state-of-the-art, yet limited by the absence of a principled approach for well-posed recovery that we begin to establish here. Original image: Tom Shlesinger. We define the task of recovering inherent scene radiance as (spectral) color recovery . This task is distinct from color enhancement —which is inherently subjective and aims to produce visually pleasing images—as well as from RGB-based reconstruction and restoration methods, which seek colorimetric consistency but are not constrained to recover underlying physical radiance. In contrast, our objective is physically grounded recovery of inherent radiance itself, which indirectly also restores scene visibility (Fig. 2 ). Consequently, a color-recovered image serves as the canonical input that brings the scene into the clear air domain (potentially after whitebalancing), for which the majority of computer vision and machine learning algorithms are optimized. We therefore argue that color recovery is the foundational objective in underwater computer vision, and more broadly in imaging in scattering media, as its successful execution restores the physical signal for all downstream tasks. When is color recovery possible? Surprisingly, despite decades of research into underwater imaging, this fundamental question has remained both unasked and unanswered. In this work, we initiate the first theoretical exploration of the conditions under which the color recovery problem can be well-posed. When color recovery is possible, the underwater camera becomes a scientific measurement device as it reflects physical reality most accurately. Color becomes a reliable physical cue, potentially enabling better segmentation, identification, classification, faster expert annotation, and improved learning. Restored visibility strengthens geometric and motion-based tasks, including visual SLAM, obstacle avoidance, structure-from-motion, and autonomous navigation, making robotic and diver operations more robust and accurate. Furthermore, because recovering inherent radiance necessarily entails solving for the medium’s optical parameters, each image becomes an environmental observation. Thus, placing color recovery on a principled basis would allow the field to move from subjective image enhancement to objective measurements with scientific precision. The data bottleneck Computer vision is no stranger to ill-posed problems: image denoising, monocular depth estimation and intrinsic image decomposition are but a few examples, many of which admit practically useful solutions. Even handwritten digit recognition, where convolutional neural networks saw their first successful real-world application [ 13 ] , is not merely ill-posed, but is ill-defined—lacking a formal specification of what constitutes a valid instance of a given digit. The groundbreaking idea making all of these problems tractable is that a predictive model can be “trained” on labeled examples, which is often much easier than designing a explicit or handcrafted method. Today, we know this as supervised learning, though the roots of this idea can be traced to Newton’s work on regression analysis. As such, the ill-posedness of color recovery in scattering media is therefore not a priori impossible to circumvent. Just like monocular depth estimation uses commonly occurring environmental features to infer the 3D geometry of the scene, analogous features can, in principle, be used to infer the color distortions caused by a medium such as water. The ideal dataset for this would then be a diverse collection of pairs of images, with and without the medium, wherein lies the main issue for the underwater case: Not only does this dataset not exist in the present day, but even in the foreseeable future, it is hard to imagine how it could be practically collected in the first place, given the vast size of the global ocean, the extreme difficulty and expense of working at sea, and the enormous variability in optical conditions across space and time. The remaining alternative of using synthetic data, although powerful in principle, does not come without serious drawbacks of its own. Light propagation in a scattering medium like water is ultimately a physical process, and to accurately simulate the apparent radiances in a scene requires accurately modeling the outcome of this process. The radiative transfer equation (RTE) is designed to do just that, and is even used in astronomy to model light in stars. Being a difficult partial differential equation to solve analytically, rendering even a moderately complex scene often requires computing the paths of millions of individual photons using Ray Tracing or Monte Carlo simulations [ 22 , 21 ] . While physically accurate, these methods require immense computational resources to converge, making them challenging to scale for the massive datasets required by machine learning. Furthermore, off-the-shelf rendering tools offer no shortcut; robust physically-based engines like Mitsuba [ 20 ] , Blender [ 6 ] , or LuxCoreRender [ 15 ] are optimized for atmospheric transport and cannot be readily tuned to the unique optical complexities of the aquatic medium. Adapting these engines to the ocean requires the manual implementation of volumetric phase functions and scattering coefficients. However, the field currently lacks a consensus on which models best represent physical reality or generalize across diverse water types [ 29 ] . Scaling considerations aside, for a computer vision model to adequately generalize from synthetic to real data, the distribution of synthetic samples should generally be comparable to that of real samples. In rendering underwater scenes, this would mean that the scene reflectances are seeded in a realistic manner. But to know which reflectances are common in real underwater environments is to have access to a richer version of the very dataset that we are trying to create. Given these challenges, progress requires a different approach: establishing when color recovery is mathematically well-posed. We provide the first theoretical insights of the conditions under which inherent scene radiance can be uniquely recovered from image data. Rather than relying on empirical validation or proposing new algorithms, we analyze the structure of the problem itself and identify sufficient conditions for eliminating its fundamental ambiguities. Color as a quantitative signal A scientific measurement device is only as good as its ability to control measurement errors. Underwater color, despite its potential as a quantitative signal, remains largely untapped because the sources of error in current methods are poorly understood. With this in mind, we identify five common method features that make error estimation difficult in absence of adequate ground-truth data: (i) supervised learning , in which models inherit biases from non-physical training data, including subjectively color-corrected imagery; (ii) arbitrary hyperparameters , which risk introducing uncontrolled errors without proper validation; (iii) non-convex optimization , making outcomes dependent on the initial guesses and therefore implicitly on arbitrary choices; (iv) ill-posedness , when multiple solutions might fit the observed data; (v) modeling in a non-physical domain , such as RGB, which neglects the spectral information loss imposed by camera sensitivities. A survey of typical methods that operate in the RGB domain (e.g., [ 5 , 3 , 27 , 33 , 32 , 9 , 12 , 31 , 30 , 11 , 14 , 18 ] show that they all have arbitrary hyperparameters and risk being ill-posed, and the majority also depend on non-convex optimization (excluding [ 5 ] ) and supervised learning (specifically [ 27 , 32 , 9 , 12 , 31 , 30 ] ). Preliminaries We denote real numbers by \mathbb{R} . Let \Lambda\subset\mathbb{R} be the wavelength interval defining the visible spectrum, typically \Lambda=[400,700] nm, and let \lambda\in\Lambda denote individual wavelengths. Spectral quantities like inherent and apparent radiances or camera sensitivities are modeled as square-integrable functions \Lambda\to\mathbb{R} forming the vector space denoted by \mathcal{L}^{2}(\Lambda) . Camera projection When a spectral radiance signal F is captured by a camera with spectral sensitivities S_{j} , where j runs over all channels, the resulting pixel intensities are given by \displaystyle P_{j}=\textstyle\int_{\Lambda}FS_{j}\ \mathrm{d}\lambda\ . (1) With only finitely many channels, perfect recovery of F from the P_{j} and the S_{j} is impossible. Indeed, let \displaystyle\langle S_{j}\rangle_{j}^{\perp}=\{U\in\mathcal{L}^{2}(\Lambda)\mid\textstyle\int_{\Lambda}US_{j}\ \mathrm{d}\lambda=0\text{ for all }j\} denote the orthogonal complement for the \mathbb{R} -span of the S_{j} . For any \Delta F\in\langle S_{j}\rangle_{j}^{\perp} , \displaystyle P_{j}=\textstyle\int_{\Lambda}FS_{j}\ \mathrm{d}\lambda=\textstyle\int_{\Lambda}(F+\Delta F)S_{j}\ \mathrm{d}\lambda\ , so perturbations by \langle S_{j}\rangle_{j}^{\perp} are “invisible” to the camera, making the recovery of F ill-posed. Such perturbations may, however, be interpreted as “details of lesser importance”, depending on the camera. By analogy, if the S_{j} were low-frequency harmonics on \Lambda , as in Fourier series, then the \langle S_{j}\rangle_{j}^{\perp} -component of F would correspond to the high-frequency “details” often neglected in engineering. Whether this interpretation is reasonable depends on the application, though a sufficiently well-behaved choice of the S_{j} will make it so, e.g. a hyperspectral camera. To compute the canonical “low-detail” approximation of F , take the unique decomposition \displaystyle F=\textstyle\sum_{k}F_{k}S_{k}+F^{\perp} (2) with F_{k}\in\mathbb{R} and F^{\perp}\in\langle S_{j}\rangle_{j}^{\perp} , which is possible because \mathcal{L}^{2}(\Lambda)=\langle S_{j}\rangle_{j}\oplus\langle S_{j}\rangle_{j}^{\perp} . Inserting ( 2 ) into ( 1 ) yields \displaystyle P_{j} \displaystyle=\textstyle\int_{\Lambda}(\textstyle\sum_{k}F_{k}S_{k}+F^{\perp})S_{j}\ \mathrm{d}\lambda \displaystyle=\textstyle\sum_{k}F_{k}\textstyle\int_{\Lambda}S_{j}S_{k}\ \mathrm{d}\lambda\ , which defines a linear system in the coefficients F_{k} . In matrix notation: \displaystyle[P_{j}]_{j}=[\textstyle\int_{\Lambda}S_{j}S_{k}\mathrm{d}\lambda]_{j,k}[F_{k}]_{k}\ . (3) The Gram matrix [\textstyle\int_{\Lambda}S_{j}S_{k}\mathrm{d}\lambda]_{j,k}\in\mathbb{R}^{n\times n} , where n is the number of channels, is non-singular whenever the S_{j} are linearly independent. In any case, \textstyle\sum_{k}F_{k}S_{k}=F-F^{\perp} can be recovered provided the S_{j} are known, and this is the best approximation of F that our camera can provide in the standard norm on \mathcal{L}^{2}(\Lambda) . Fig. 3: From spectra to pixels. The apparent scene radiance F is mapped to discrete pixel values through integration against the camera’s spectral sensitivities S_{R},S_{G},S_{B} . This projection compresses infinite-dimensional spectral information to a finite number channel values, imposing a fundamental limitation on color recovery. Image formation Along any line of sight from a scene point to the camera, the inherent radiance gives rise to the apparent radiance at the camera. Image formation in color recovery relates these two quantities. We now give a derivation of a tractable simplification of the RTE from [ 23 ] . Let z denote the distance along the line of sight towards the camera, and let Q(z) denote the corresponding radiance. If z_{0}=0 and the camera is at z_{1} , then Q(z_{0}) is the inherent radiance while Q(z_{1}) is the apparent radiance. The instantaneous radiance gain Q^{+}(z) is due to scattering from other directions and potential emission, while the radiance loss Q^{-}(z) is due to absorption and out-of-path scattering: Q^{-}=aQ+bQ=cQ , where a , b and c=a+b are the coefficients of beam absorption, scattering, and attenuation, respectively. The RTE states that \frac{\partial Q}{\partial z}=Q^{+}-cQ , which is a first order inhomogeneous linear equation solved by \displaystyle Q(z)=e^{-cz}Q(0)+\textstyle\int_{0}^{z}e^{c(\tilde{z}-z)}Q^{+}(\tilde{z})\mathrm{d}\tilde{z}\ , (4) assuming c is constant in z . The relation between Q(z) and Q(0) is thus determined by Q^{+} , whose faithful modeling would require accounting for interactions between all the paths that light could take, since radiance scattered out of one path contributes to another. To circumvent these complications, which would otherwise require sophisticated numerical methods, we adopt an extreme but common simplifying assumption: The instantaneous radiance gain Q^{+} in any line of sight terminating at the camera is independent of z and the line of sight itself. Equation ( 4 ) then reduces to: \displaystyle Q(z) \displaystyle=e^{-cz}Q(0)+Q^{+}e^{-cz}\textstyle\int_{0}^{z}e^{c\tilde{z}}\mathrm{d}\tilde{z} \displaystyle=e^{-cz}Q(0)+(1-e^{-cz})\frac{Q^{+}}{c}\ . For notational convenience, we write F=Q(z) , L=Q(0) and B=Q^{+}/c , yielding: \displaystyle F=e^{-cz}L+(1-e^{-cz})B\ . (5) In color recovery, we treat ( 5 ) as relating F and L through the (typically unknown) medium parameters c and B , the latter often called backscatter at infinity . One physical interpretation is a single particle suspended in water, reflecting light in all directions (Figure 4 ), though constancy of the instantaneous radiance gain Q^{+} requires an additional assumption: photons leaving the particle can undergo at most one scattering event, so none initially outside the line of sight can enter it. Consequently, Q^{+} is determined solely by “ambient” photons. If their contribution is spatially constant, the line of sight must be horizontal under uniform surface illumination. Under this interpretation, the pixel corresponding to the line of sight towards the particle satisfies ( 5 ), though neighboring pixels do not, due to the photons originating at the particle that scatter towards the camera along different paths. The resulting weak halo, known as forward scatter , must be negligible for the model to hold precisely. The model is therefore invoked for simplicity rather than accuracy, though even stronger simplifications have proven effective [ 3 , 14 ] . Fig. 4: The physics of underwater image formation. Inherent radiance reflected from a single particle (yellow arrows) is exponentially attenuated due to absorption and scattering events (blue circles). In addition, path radiance (backscatter) generated by scattering along line-of-sight introduces the occluding haze (blue arrows). The apparent scene radiance incident at the camera, and consequently the recorded pixel intensities represent the combination of attenuated scene radiance with backscatter. Ill-posedness The ill-posedness of color recovery is understood by characterizing how unobserved quantities may vary without affecting measurements. Suppose the camera allows perfect reconstruction of the apparent radiance F . Even then, if the medium parameters c and B are unknown, the inherent radiance L fails to be uniquely determined, since \displaystyle L=e^{cz}F+(1-e^{cz})B\ , which yields a valid L for any choice of c and B . Relaxing the assumption of a perfect camera, even perfect knowledge of c and B is insufficient. Writing F=\sum_{j}F_{j}S_{j}+F^{\perp} with F^{\perp}\in\langle S_{j}\rangle_{j}^{\perp} , we then have \displaystyle L=e^{cz}\textstyle\sum_{j}F_{j}S_{j}+(1-e^{cz})B+e^{cz}F^{\perp}\ , where F^{\perp} is free to vary within \langle S_{j}\rangle_{j}^{\perp} . Consequently, progress requires additional assumptions. For simplicity, we assume a perfect camera, i.e., that \langle S_{j}\rangle_{j}^{\perp}=\{0\} , treating “invisible” high-resolution spectral details as negligible. When higher accuracy is needed, a more capable camera (e.g. hyperspectral) may be required. Even with a perfect camera, however, the inherent ill-posedness due to unknown c and B remains. Well-posed color recovery In the preliminaries, we justified making the assumption of an ideal lossless camera. Thus, for each pixel i , we assume full knowledge of the apparent radiance \displaystyle F_{i}=e^{-cz_{i}}L_{i}+(1-e^{-cz_{i}})B\ , (6) where L_{i} and z_{i} denote inherent radiance and distance. The distances z_{i} are assumed known, while c,B and L_{i} are unknown. For any given i , this is one equation with three unknowns, and L_{i} cannot be recovered from a single pixel. Considering all pixels jointly does not resolve the ambiguity: any “guess” of c and B is equally valid and yields a different estimate of \displaystyle L_{i}=e^{cz_{i}}F_{i}+(1-e^{cz_{i}})B\ , so further assumptions are necessary. Rather than constraining c and B , we follow prior work on underwater imaging and assume regularities in the inherent radiances. The dark channel prior [ 10 ] , for instance, postulates that natural scenes contain regions where the inherent radiance is close to zero. Such regions are readily identified, and their apparent color constrains the medium. More generally, whenever certain knowledge about a group of pixels makes color recovery well-posed, we term this group a recovery pattern . We identify six kinds of such patterns, including the a variant of the dark channel prior, with full derivations given in the Supplementary Materials. In contrast to existing work, however, we stay in the spectral domain, and derive explicit formulae for c and B in each case, avoiding potentially non-convex optimization. Recovery patterns A recovery pattern , as we define it, is a group of pixels with known relations that render color recovery well-posed across the entire image. A non-example is a single dark pixel with apparent radiance F=(1-e^{-cz})B , where the two unknowns c and B fail to be uniquely determined by this single equation. Two such pixels, however, with F_{i}=(1-e^{-cz_{i}})B and z_{1}\neq z_{2} , do suffice, since \displaystyle\frac{F_{1}}{F_{2}}=\frac{1-e^{-cz_{1}}}{1-e^{-cz_{2}}}\doteq\Phi_{z_{1},z_{2}}(c)\ . The function \Phi_{z_{1},z_{2}} is injective when z_{1}\neq z_{2} , making c , and consequently also B , uniquely determined by F_{1} and F_{2} . This recovery pattern consisting of two dark pixels may be regarded as a minimally viable case of the dark channel prior [ 10 ] . Using similar reasoning, we identify six kinds of recovery patterns in total (Table I ; see Supplementary Text for derivations). These patterns provide sufficient conditions for well-posed recovery, and should not be interpreted as exhaustive. Other cross-pixel relationships may also render recovery well-posed, and their characterization remains an open question. Furthermore, these patterns represent idealized mathematical conditions; practical implementation requires developing robust methods to detect such patterns in noisy, real-world images—an algorithmic challenge beyond the scope of this theoretical work. Recovery patterns # Diagram Definition Medium Interpretation 1 z_{1} 0 z_{2} L_{1}=L_{2}=0\newline z_{1}\neq z_{2} c\,=\Phi_{z_{1},z_{2}}^{-1}(\frac{F_{1}}{F_{2}})\newline B=\frac{F_{1}}{1-e^{-cz_{1}}}=\frac{F_{2}}{1-e^{-cz_{2}}} Two dark pixels (e.g., perfect black or shadowed) at different depths, Fig. 5 ). 2 z_{1} L z_{2} z_{3} L_{1}=L_{2}=L_{3}\newline z_{1}