Fourier-Based 3D Multistage Transformer for Aberration Correction in Multicellular Specimens
Thayer Alshaabi 1,2 , Daniel E. Milkie 1 , Gaoxiang Liu 2 , Cyna Shirazinejad 2 , Jason L. Hong 2 , Kemal Achour 2 , Frederik Görlitz 2 , Ana Milunovic-Jevtic 2 , Cat Simmons 2 , Ibrahim S. Abuzahriyeh 2 , Erin Hong 2 , Samara Erin Williams 2 , Nathanael Harrison 2 , Evan Huang 2 , Eun Seok Bae 2 , · 2025
Read Fourier-Based 3D Multistage Transformer for Aberration Correction in Multicellular Specimens by Thayer Alshaabi 1,2 , Daniel E. Milkie 1 , Gaoxiang Liu 2 , Cyna Shirazinejad 2 , Jason L. Hong 2 , Kemal Achour 2 , Frederik Görlitz 2 , Ana Milunovic-Jevtic 2 , Cat Simmons 2 , Ibrahim S. Abuzahriyeh 2 , Erin Hong 2 , Samara Erin Williams 2 , Nathanael Harrison 2 , Evan Huang 2 , Eun Seok Bae 2 , (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12593.txt.
Introduction
Alshaabi and a large team present in twenty twenty-five a learning system that fixes blur in microscope images of living tissue without extra hardware. Tissue itself bends light, which softens detail and contrast. Standard fixes use sensors and mirrors that are costly, complex, and slow over large fields.
Their system uses a staged vision transformer that works on frequency patterns rather than raw pixels. It infers the blur pattern and restores sharp images with far less computing cost, training time, and memory than rival designs. Tests on live gene-edited zebrafish embryos showed it corrected blur that varied across the field. Correction worked either through a mirror device or by cleaning images after capture.
Catalog record
| CREATOR | Thayer Alshaabi 1,2 , Daniel E. Milkie 1 , Gaoxiang Liu 2 , Cyna Shirazinejad 2 , Jason L. Hong 2 , Kemal Achour 2 , Frederik Görlitz 2 , Ana Milunovic-Jevtic 2 , Cat Simmons 2 , Ibrahim S. Abuzahriyeh 2 , Erin Hong 2 , Samara Erin Williams 2 , Nathanael Harrison 2 , Evan Huang 2 , Eun Seok Bae 2 , |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12593
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fourier-Based 3D Multistage Transformer for Aberration Correction in Multicellular Specimens Thanks: ∗ * Corresponding authors: [email protected], [email protected], [email protected]. Thayer Alshaabi 1,2 , Daniel E. Milkie 1 , Gaoxiang Liu 2 , Cyna Shirazinejad 2 , Jason L. Hong 2 , Kemal Achour 2 , Frederik Görlitz 2 , Ana Milunovic-Jevtic 2 , Cat Simmons 2 , Ibrahim S. Abuzahriyeh 2 , Erin Hong 2 , Samara Erin Williams 2 , Nathanael Harrison 2 , Evan Huang 2 , Eun Seok Bae 2 , Alison N. Killilea 2 , David G. Drubin 2 , Ian A. Swinburne 2 , Srigokul Upadhyayula 2,3,4 , Eric Betzig 1,2,5,6 Abstract . † † footnotetext: Model code . Source code for training and evaluation (and all pretrained models) are available at https://github.com/cell-observatory/aovift . Synthetic data generator . Code for simulating beads is available at https://github.com/cell-observatory/beads_simulator . Package . Docker image is available at https://github.com/cell-observatory/aovift/pkgs/container/aovift . † † footnotetext: Contributions . T.A. designed models, developed training pipelines and evaluation benchmarks. D.E.M. designed the preprocessing algorithms and developed the microscope software for the imaging experiments. T.A. and D.E.M. designed Fourier embedding, and developed the synthetic data generator for training and validation. C. Shirazinnejad, C. Simmons, I.S.A., S.E.W., N.H., E. Hong, E. Huang, E.S.B., A.N.K., D.G.D., and I.A.S. generated the zebrafish reagents. A.N.K., prepared the cultured SUM159 cells. C. Shirazinnejad, J.L.H., K.A., and A.M.J. prepared samples. G.L. and J.L.H. performed the imaging experiments with zebrafish. J.L.H. and K.A. performed the imaging experiments with cells. G.L., K.A. and F.G. performed the imaging experiments with beads. T.A., D.E.M., and S.U. performed analysis and prepared figures. T.A. wrote the manuscript with input from all co-authors. D.E.M., E.B. and S.U. edited the manuscript. T.A., E.B. and S.U. supervised the project. High-resolution tissue imaging is often compromised by sample-induced optical aberrations that degrade resolution and contrast. While wavefront sensor-based adaptive optics (AO) can measure these aberrations, such hardware solutions are typically complex, expensive to implement, and slow when serially mapping spatially varying aberrations across large fields of view. Here, we introduce AOViFT (Adaptive Optical Vision Fourier Transformer)—a machine learning-based aberration sensing framework built around a 3D multistage Vision Transformer that operates on Fourier domain embeddings. AOViFT infers aberrations and restores diffraction-limited performance in puncta-labeled specimens with substantially reduced computational cost, training time, and memory footprint compared to conventional architectures or real-space networks. We validated AOViFT on live gene-edited zebrafish embryos, demonstrating its ability to correct spatially varying aberrations using either a deformable mirror or post-acquisition deconvolution. By eliminating the need for the guide star and wavefront sensing hardware and simplifying the experimental workflow, AOViFT lowers technical barriers for high-resolution volumetric microscopy across diverse biological samples. 1. Introduction Figure 1. AOViFT workflow . A . AOViFT correction. An aberrated 3D volume is preprocessed and cast into a Fourier Embedding, which is passed to a 3D vision transformer model to predict the detection wavefront. A deformable mirror (DM) compensates for this aberration, enabling acquisition of a corrected volume. B . The Fourier embedding, \mathcal{E} . The Fourier Transform of the 3D volume is embedded into a lower space ( \mathcal{E}\in\mathbb{R}^{\ell\times d\times d} ), consisting of 3 amplitude planes ( \alpha_{1},\alpha_{2},\alpha_{3} ) and 3 phase planes ( \varphi_{1},\varphi_{2},\varphi_{3} ). C . AOViFT model. The Fourier embedding is input to a dual-stage 3D vision transformer model. At each stage, the \ell Fourier planes are tiled into k patches (Patchify), applying a radially encoded positional embedding to each patch. These patches are passed through n Transformer layers. At the end of each stage, a residual connection is added, and the patches are merged back to the shape matching the stage input (Merge patches). After all stages, the resulting patches are pooled (GlobalAvgPool) and connected with a dense layer to output the z Zernike coefficients. Figure 2. Comparison of different state-of-the-art architectures when applied to 3D aberration sensing. A . Total number of trainable parameters. B . Maximum predictions per second, using a batch size of 1024 on a single A100 GPU. Higher values are better. C . Training time on eight H100 GPUs. D . Median \lambda RMS residuals over 10K test samples after one correction, with aberrations ranging between 0.2\lambda to 0.4\lambda , simulated with 50K to 200K integrated-photons. E–F . Median \lambda RMS residuals using our Small model for a single bead over a wide range of SNR. G–H . Median \lambda RMS residuals using our Small model for several beads (up to 150 beads), simulated at photon levels from 50K to 200K per bead. Lower values are better for all performance indicators listed here, except for B . As we peer deeper into living organisms to reveal their inner workings, our view is increasingly compromised by sample-induced optical aberrations. Numerous AO methods exist to compensate for these by using a wavefront shaping device that responds to a measurement of sample-induced aberration ( 1 ) . These methods differ in their complexity, generality, robustness, and practicality. In our lab, dependable success was had using a Shack-Hartmann (SH) sensor to measure the aberrations imparted on a guide star (GS) created by two-photon excitation (TPE) fluorescence within the specimen ( 2 ) , and we have used this approach extensively in adaptive optical lattice light sheet microscopy (AO-LLSM) to study four dimensional (4D) subcellular dynamic processes within the native environment of whole multicellular organisms ( 3 ) . Several recent approaches dispense with the cost and complexity of hardware-based wavefront measurement in favor of directly inferring aberrations from the microscope images themselves via machine learning (ML) ( 4 , 5 , 6 , 7 , 8 ) (Supplementary Table S1 ). Based on our experience with a variety of specimens, any ML-AO approach suitable for AO-LLSM must meet the following specifications: Speed: To maximize the range of spatiotemporal events that can be visualized, the time for the ML model to infer the aberrations across any volume should be less than the time needed to image it—typically a few seconds in LLSM for a volume that encompasses a handful of cells. Robustness: The model must accurately predict the vast majority of aberrations encountered in practice—for AO-LLSM in zebrafish embryos, typically up to 5\lambda peak-to-valley (P-V) in any combination of the first 15 Zernike modes ( Z_{0}^{0} through Z_{4}^{\pm 4} Supplementary Fig. S1 ). Accuracy: The method should be able to recover close to the theoretical 3D resolution limits of the microscope, regardless of the distribution of spatial frequencies within the specimen. Noninvasiveness: The method should provide accurate correction without unduly depleting the fluorescence photon budget within the specimen or perturbing its native physiology. As none of the aforementioned ML-AO methods meet all these specifications, we endeavored to create one better suited to the needs of AO-LLSM. Our baseline model architecture, selected from an ablation study (Appendix A , Supplementary Fig. S2 – S4 ), contains two transformer stages with patches of 32 and 16 pixels, respectively (Fig. 1 c). Priors can greatly improve the performance of any ML approach. For our method, we depend on the prior that each isoplanatic subvolume (i.e., having the same aberration) within the larger volume of interest contains one or more fluorescent puncta of true sub-diffractive size. Here, we introduce these by using genome-edited