From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy
Feng He, Guodong Tan, Qiankun Li, Jun Yu, Quan Wen University of Science and Technology of China [email protected], [email protected], [email protected]@ustc.edu.cn, [email protected]*Corresponding Author · 2025
Read From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy by Feng He, Guodong Tan, Qiankun Li, Jun Yu, Quan Wen University of Science and Technology of China [email protected], [email protected], [email protected]@ustc.edu.cn, [email protected]*Corresponding Author (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22577.txt.
Introduction
Light field microscopy captures a whole volume in one exposure, helping neuroscientists film living brain activity. Feng He, Guodong Tan, Qiankun Li, Jun Yu, and Quan Wen tackle weak reconstruction methods for a large field version of that tool in a 2025 paper.
They build XLFM-Zebrafish, a large benchmark dataset and test suite for this imaging style. Their learning task hides some camera views and predicts them, teaching angle aware structure with less labeled data. A new loss checks that predicted volumes, when passed through the known optics, match real projections. Their method improves a standard quality score by several percent over baselines.
Catalog record
| CREATOR | Feng He, Guodong Tan, Qiankun Li, Jun Yu, Quan Wen University of Science and Technology of China [email protected], [email protected], [email protected]@ustc.edu.cn, [email protected]*Corresponding Author |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22577
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy Feng He, Guodong Tan, Qiankun Li, Jun Yu, Quan Wen University of Science and Technology of China [email protected], [email protected], [email protected]@ustc.edu.cn, [email protected]*Corresponding Author Abstract Light field microscopy (LFM) has become an emerging tool in neuroscience for large-scale neural imaging in vivo, with XLFM (eXtended Light Field Microscopy) notable for its single-exposure volumetric imaging, broad field of view, and high temporal resolution. However, learning-based 3D reconstruction in XLFM remains underdeveloped due to two core challenges: the absence of standardized datasets and the lack of methods that can efficiently model its angular–spatial structure while remaining physically grounded. We address these challenges by introducing three key contributions. First, we construct the XLFM-Zebrafish benchmark, a large-scale dataset and evaluation suite for XLFM reconstruction. Second, we propose Masked View Modeling for Light Fields (MVM-LF), a self-supervised task that learns angular priors by predicting occluded views, improving data efficiency. Third, we formulate the Optical Rendering Consistency Loss (ORC Loss), a differentiable rendering constraint that enforces alignment between predicted volumes and their PSF-based forward projections. On the XLFM-Zebrafish benchmark, our method improves PSNR by 7.7% over state-of-the-art baselines. Code and datasets are publicly available at: https://github.com/hefengcs/XLFM-Former. 1 Introduction Light Field Microscopy (LFM) has emerged as a crucial technique for rapid volumetric imaging of nervous systems LFM ; LFM1 ; LFM2 . Notably, eXtended Light Field Microscopy (XLFM) XLFM , due to its graceful balance between speed, scale and resolution, is considered one of the most suitable LFM techniques for large-scale neural activity recording in several model organisms, including fish and mouse LFM3 . XLFM offers several advantages: 1)XLFM enables single-exposure acquisition of complete light field information at 100 Hz, whereas conventional microscopy techniques (e.g., two-photon microscopy 2p , light-sheet microscopy sheet ) require sequential layer-by-layer scanning, making it challenging to capture sub-second large-scale neural dynamics simultaneously. 2) The XLFM system incorporates a point spread function (PSF) that is approximately spatially invariant. Consequently, the reconstruction of volumes through 3D deconvolution is free from artifacts. 3) The rapid speed of XLFM allows real-time observation of large-scale population neural activity in vivo. Integrating volumetric imaging with optogenetic manipulation op1 ; op2 will enable optical brain-machine interface, namely closed-loop optical interrogation of brain-wide activity in both immobilized ShangNN and freely behaving animals XLFM ; All-optical . Despite the promise of eXtended Light Field Microscopy (XLFM) for rapid volumetric neural imaging, progress in learning-based 3D reconstruction remains limited not only due to the unique physics of XLFM, but also because of a lack of standardized datasets and evaluation protocols. First, XLFM data differs fundamentally from conventional image data: each frame encodes a dense angular sampling of the 3D scene via a microlens array, creating highly entangled multi-view observations. Traditional convolutional models struggle to model these angular correlations and view-dependent cues effectively. In addition, raw XLFM acquisitions are abundant, producing high-quality volumetric ground truth (e.g., via Richardson–Lucy deconvolution RLD ) is computationally expensive. This makes supervised learning pipelines costly at scale. Moreover, there is currently no public benchmark dataset or reproducible evaluation protocol for XLFM reconstruction. As a result, comparisons across methods are often anecdotal, and progress in the field is fragmented. Finally, most existing approaches overlook the wave-optical nature of the XLFM forward model. Without incorporating physics-guided constraints such as point spread function (PSF) priors, reconstructions may be visually plausible but physically inconsistent. Figure 1 : Our pretraining pipeline for XLFM. The raw light field acquired from the microscope is separated into 27 distinct viewpoints based on physical coordinates. With a 70% probability, we randomly mask a subset of these viewpoints and task the model with reconstructing them. The training is supervised by an \ell_{2} loss comparing the predicted and ground-truth views. To address the unique challenges of XLFM reconstruction, we revisit the problem not merely as a supervised regression task, but as a structured prediction problem grounded in physical optics, spatial geometry, and data asymmetry. Our design of XLFM-Former is driven by four key insights: 1) XLFM reconstruction inherently requires long-range dependency modeling across a large volumetric field-of-view with densely entangled angular observations. While increasing the receptive field in convolutional models like U-Net can improve performance, it also incurs steep memory overhead, scaling linearly with spatial resolution and depth. Alternative global modeling strategies such as Fourier neural operators introduce even higher complexity, often transforming real-valued tensors into complex-valued representations that exceed the memory capacity of a single 80GB GPU. In contrast, we adopt a Swin Transformer encoder with hierarchical windowed attention, which efficiently captures both local and global dependencies with significantly reduced memory costs that making it a natural fit for large-scale 3D light field modeling. 2) Unlike conventional image data, XLFM views are not independent. They exhibit occlusion patterns, spatial redundancy, and angular continuity, much like dependencies observed in natural language or multiview stereo systems. Modeling these view-wise interactions is essential for resolving fine 3D structures from ambiguous projections. We argue that the view, not the pixel, should be treated as the atomic modeling unit. Our MVM-LF pretraining task (Figure 1 ) reconstructs masked views from their angular neighbors, allowing the model to internalize structural priors specific to the XLFM sampling pattern. Figure 2 : The multi-view images used for pretraining an encoder model. For each triplet, we show the masked image (left), our MVM-LF regenerated image (middle), and the ground-truth (right). The masked regions are generated by applying the binary mask complement to the original image. Figure 3 : Regeneration of XLFM light field images via MVM-LF. The model can still accurately predict the view under appropriate occlusion, indicating that it has learned the global view relationship. Excessive occlusion (90%) causes prediction to crash, indicating that MVM-LF requires a reasonable occlusion ratio to balance information loss and network learning ability. 3) The data economics of XLFM present a natural motivation for self-supervised learning. Capturing light field images is fast, inexpensive, and non-destructive, but generating high-quality volumetric annotations (e.g., via Richardson–Lucy deconvolution) is computationally intensive. Our pretraining strategy allows the model to scale with data availability while reducing reliance on expensive labels, improving both generalization and transferability to new imaging settings. Our pretraining strategy masks a random subset of angular views and tasks the model with reconstructing them. This view-level self-supervision enables the model to scale with unlabeled XLFM data while improving generalization and transferability to unseen imaging conditions. Figure 2 illustrates the model’s ability to reconstruct masked views across diverse zebrafish samples, while Figure 3 shows its robustness and limitations under varying occlusion levels. 4) Even visually accurate reconstructions may diverge from the underlying physics of light propagation. Models trained purely on pixel-level losses often hallucinate plausible but optically inconsistent structures undermining the scientific validity of the output. To enforce physical plausibility, we introduce the Physically Optical Rendering Consistency Loss (ORC Loss), which ensures that reconstructed volumes, when passed through the known point spread function (PSF) of the imaging system, yield a synthetic light field that matches the observed measurements. This alignment with the optical forward model constrains the network to produce predictions that are both data-aligned and physics-consistent. Our main contributions are summarized as follows: \arabicenumi⃝ A standardized benchmark for XLFM reconstruction. We construct the first