Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning
SiYeoul Lee, SeonHo Kim, Minkyung Seo, SeongKyu Park, Salehin Imrus, Kambaluru Ashok, DongEon Lee, Chunsu Park, SeonYeong Lee, Jiye Kim, Jae-Heung Yoo, MinWoo Kim · 2025
Read Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning by SiYeoul Lee, SeonHo Kim, Minkyung Seo, SeongKyu Park, Salehin Imrus, Kambaluru Ashok, DongEon Lee, Chunsu Park, SeonYeong Lee, Jiye Kim, Jae-Heung Yoo, MinWoo Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03505.txt.
Introduction
SiYeoul Lee and eleven co-authors introduce a learning network that lifts free-hand three-dimensional photoacoustic and ultrasound reconstruction, in a paper from twenty twenty-five. Handheld imaging has a narrow field of view and struggles with complex shapes. Free-hand reconstruction aligns sequential two-dimensional images into volumes. Gauging the probe's motion without outside place sensors is the hard part.
Their network, MoGLo-Net, adapts self-attention to use image regions rich in detail, such as developed speckle or bright tissue. A patch-wise correlation step builds a volume tied closely to scanning motion. A custom loss function keeps learning robust and low-bias. Tests showed results beyond current state-of-the-art methods, in numbers and in kind. The team took reconstruction beyond plain ultrasound volumes. Doppler ultrasound and photoacoustic imaging were also shown, and widened its clinical reach.
Catalog record
| CREATOR | SiYeoul Lee, SeonHo Kim, Minkyung Seo, SeongKyu Park, Salehin Imrus, Kambaluru Ashok, DongEon Lee, Chunsu Park, SeonYeong Lee, Jiye Kim, Jae-Heung Yoo, MinWoo Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-03505
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Enhancing Free-hand 3D Photoacoustic and Ultrasound Reconstruction using Deep Learning SiYeoul Lee, SeonHo Kim, Minkyung Seo, SeongKyu Park, Salehin Imrus, Kambaluru Ashok, DongEon Lee, Chunsu Park, SeonYeong Lee, Jiye Kim, Jae-Heung Yoo, MinWoo Kim Abstract This study introduces a motion-based learning network with a global-local self-attention module ( MoGLo-Net ) to enhance 3D reconstruction in handheld photoacoustic and ultrasound (PAUS) imaging. Standard PAUS imaging is often limited by a narrow field of view and the inability to effectively visualize complex 3D structures. The 3D freehand technique, which aligns sequential 2D images for 3D reconstruction, faces significant challenges in accurate motion estimation without relying on external positional sensors. MoGLo-Net addresses these limitations through an innovative adaptation of the self-attention mechanism, which effectively exploits the critical regions, such as fully-developed speckle area or high-echogenic tissue area within successive ultrasound images to accurately estimate motion parameters. This facilitates the extraction of intricate features from individual frames. Additionally, we designed a patch-wise correlation operation to generate a correlation volume that is highly correlated with the scanning motion. A custom loss function was also developed to ensure robust learning with minimized bias, leveraging the characteristics of the motion parameters. Experimental evaluations demonstrated that MoGLo-Net surpasses current state-of-the-art methods in both quantitative and qualitative performance metrics. Furthermore, we expanded the application of 3D reconstruction technology beyond simple B-mode ultrasound volumes to incorporate Doppler ultrasound and photoacoustic imaging, enabling 3D visualization of vasculature. The source code for this study is publicly available at: https://github.com/guhong3648/US3D Preprint, compiled August 24, 2026 K eywords Deep Learning, Free-hand 3D Reconstruction, Ultrasound Imaging, Photoacoustic Imaging. 1 Introduction Diagnostic ultrasound (US) is a widely employed modality for examining various organs and tissues, owing to its real-time imaging capabilites. Particularly, it serves as a preferred tool for guiding medical procedures such as biopsies and injections [ 1 ] . In addition, US plays a pivotal role in dynamically monitoring the vascular system through the Doppler effect [ 2 ] . Meanwhile, the integration of photoacoustic (PA) imaging with conventional US imaging (PAUS imaging) has been extensively studied because of its promising clinical potential. This integration aims to enhance the existing benefits of ultrasound while introducing novel capabilities for both interventional and functional imaging [ 3 ] . In the PAUS system, a handheld transducer is responsible for emitting US or/and laser pulses and receiving resulting acoustic wave signals. While this configuration allows users the flexibility to manually scan a region of interest, it is accompanied by certain limitations. Notably, the field of view is restricted, providing only a narrow 2D cross-section of the image area, which hampers the understanding of the topological 3D structures of the target. Although some transducers are specifically designed for 3D imaging, their physical dimensions often render them impractical for handheld use [ 4 ] . An alternative approach is the 3D freehand method, which involves the sequential alignment of 2D image sections acquired from freehand sweeps with a standard transducer. Users can playback a series of 2D US image frames as a video through a CINE loop, but the visualization of the 3D rendering volume or structure of the target through accumulating frames is more straightforward. This method proves ideal in clinical practice when tissue motion is static. However, a primary technical challenge of this approach lies in estimating sweeping motion for correct 3D reconstruction. Although an external positional device can be attached to the transducer, it results in making the transducer bulkier and often provides inaccurate measurements in a clinical environment due to various optical or electrical disturbances [ 5 ] . Several studies have explored the estimation of scan trajectories directly from 2D US images, eliminating the need for external sensors [ 6 , 7 , 8 ] . The key idea is to exploit and track tissue speckle patterns in US B-mode images. Recent approaches have employed deep learning (DL) frameworks to better leverage speckle features for predicting scan motion, typically defined by six parameters. Since the pioneering work of Prevost et al. [ 9 ] , which introduced the first DL model for this task, subsequent studies have proposed advanced models [ 10 , 11 , 12 , 13 , 14 ] to improve stability and accuracy. However, their overall estimation accuracies were still insufficient for clinical practice, and they were rarely validated for long elevational scan trajectories, which are common in clinical practice. The principal contribution of this study lies in proposing a novel deep learning method to elevate visualization capabilities of 3D US and PAUS system. The structure is composed of a ResNet-driven encoder [ 15 ] and an estimator with a Long Short-Term Memory (LSTM) block to extract features from given B-mode frames and estimate the motion vectors between consecutive frames, leveraging long-term memory. One part of the encoder consists of special blocks that can directly access the correlation between encoded feature maps from adjacent frames. Additionally, we designed task-specific novel global-local attention module, which effectively highlights the critical local regions for motion estimation, such as fully-developed speckle areas or high-echogenic tissue areas. Moreover, our customized loss function ensures that the network robustly learns motion without a significant distortion in specific parameter estimation. The model was rigorously evaluated using both our proprietary dataset and a publicly available dataset. Utilizing a programmable ultrasound system enabled us to assess the model’s performance across a variety of conditions, such as reductions in B-mode image speckle or the inclusion of more unprocessed data as input. Comparing deep learning methods is inherently difficult due to the fact that each model is typically designed and optimized for a specific acquisition dataset. To mitigate this issue, we also employed an publicly open dataset [ 15 ] , obtained under entirely different settings from our own. By training and testing our model on open dataset, we aimed to demonstrate that our framework is generalizable and not overly dependent on the dataset we specifically acquired. Another significant contribution involves highlighting the versatility of the panorama technique in clinical applications, extending beyond the mere compilation of B-mode images. Through the combination of US and laser transmission sequencing, hand-held free-scan PAUS system can obtain PA data as well as US data at once. Using positioning contents, we demonstrate that 3D reconstruction of vessels is available from either US power Doppler mode (PD-mode) data or PA data. To the best of our knowledge, this is the first reported application of this approach. These advancements have the potential to aid specialists in diagnosing diseases or precisely locating targets during the intervention process. 2 Related Works Traditional approaches exploited tissue speckle patterns in US B-mode images, as speckle content between successive frames tends to be preserved, even in out-of-plane motions. Since the pioneering work by Trahey et al. [ 16 ] , several studies have focused on estimating scan motion by analyzing the correlation between adjacent frames [ 6 , 7 ] . Gee et al. [ 8 ] proposed an adaptive speckle decorrelation technique, leading to improved performance. These studies have demonstrated the feasibility of sensorless free-hand US through gradual improvements in various factors such as estimation accuracy, generalizability, and less constrained scan protocols. With the rapid advancements in DL, Prevost et al. [ 9