MT3DNet: Multi-Task learning Network for 3D Surgical Scene Reconstruction
Mithun Parab ∗, Pranay Lendave ∗, Jiyoung Kim, Thi Quynh Dan Nguyen, Palash Ingle · 2024
Read MT3DNet: Multi-Task learning Network for 3D Surgical Scene Reconstruction by Mithun Parab ∗, Pranay Lendave ∗, Jiyoung Kim, Thi Quynh Dan Nguyen, Palash Ingle (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03928.txt.
Introduction
Mithun Parab and co-authors build a multi task network in a 2024 paper on surgery imaging. In keyhole surgery, live scene understanding helps guide surgeons, judge skill, and support robot teamwork. One system must find tools, outline them, guess depth, and rebuild the scene in three dimensions, all at once.
Their network, MT3DNet, learns those tasks jointly rather than in separate tools. Joint learning is hard to tune, since tasks pull the shared parts in different directions. The design tackles that tuning problem head on. A single pass over a sharp surgical image then yields labels, outlines, depth, and a 3D scene for feedback in the operating room.
Catalog record
| CREATOR | Mithun Parab ∗, Pranay Lendave ∗, Jiyoung Kim, Thi Quynh Dan Nguyen, Palash Ingle |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MT3DNet: Multi-Task learning Network for 3D Surgical Scene Reconstruction Mithun Parab ∗, Pranay Lendave ∗, Jiyoung Kim, Thi Quynh Dan Nguyen, Palash Ingle Abstract In image-assisted minimally invasive surgeries (MIS), understanding surgical scenes is vital for real-time feedback to surgeons, skill evaluation, and improving outcomes through collaborative human-robot procedures. Within this context, the challenge lies in accurately detecting, segmenting, and estimating the depth of surgical scenes depicted in high-resolution images, while simultaneously reconstructing the scene in 3D and providing segmentation of surgical instruments along with detection labels for each instrument. To address this challenge, a novel Multi-Task Learning (MTL) network is proposed for performing these tasks concurrently. A key aspect of this approach involves overcoming the optimization hurdles associated with handling multiple tasks concurrently by integrating a Adversarial Weight Update into the MTL framework, the proposed MTL model achieves 3D reconstruction through the integration of segmentation, depth estimation, and object detection, thereby enhancing the understanding of surgical scenes, which marks a significant advancement compared to existing studies that lack 3D capabilities. Comprehensive experiments on the EndoVis2018 benchmark dataset underscore the adeptness of the model in efficiently addressing all three tasks, demonstrating the efficacy of the proposed techniques. $*$ $*$ footnotetext: These authors contributed equally to this work 1 Introduction Efficient 3D reconstruction for surgical scene understanding of MIS environments is crucial during robotic surgery since it enhances precision, reliability and reduces trauma and recovery time. However, surgeons can still face workflow disruptions due to limited tactile feedback or system issues. Moreover, the complexity of surgical environments, characterized by factors such as smoke, bodily fluids, varying lighting conditions, and partial occlusions, poses significant challenges to image interpretation [ 16 ] . Hence, there arises a pressing need for 3D reconstruction of surgical scenes, it enhances surgical performance, offers real-time feedback, aids in skill assessment for novice surgeons, and enables detailed analysis of tool movements. Recognizing the limitations of single-task learning (STL) for individual tasks like semantic segmentation [ 22 ] , instance segmentation [ 13 ] , and object detection [ 4 ] in surgical scene understanding, a multi-task learning (MTL) approach is preferred that performs all three tasks concurrently, promoting computational efficiency. However, recent breakthroughs in MTL haven’t addressed depth estimation, which is crucial for 3D reconstruction, leaving a gap in achieving truly comprehensive scene understanding. There are approaches for 3D reconstruction by finding disparity maps using stereo camera setup [ 17 ] , but these approaches adds an additional camera which is not feasible in hardware and space constrained MIS procedures. Thus, this approach emphasizes on monocular depth estimation using a single camera view that can be used for 3D reconstruction of the MIS environment. The progress in comprehending surgical scenes heavily relies on the EndoVis2018 datasets, which feature annotated robotic instrument data for tasks like segmentation and detection. Despite containing seven tube-like instruments, these datasets lack monocular depth annotations for MIS settings. To address this, the authors employed the state-of-the-art STL depth estimation model, Depth Anything [ 24 ] , to generate depth maps for training the MTL model. Our Contributions: • Develop a novel multi-task framework for performing three tasks: segmentation, instrument detection, and monocular depth estimation concurrently. • Implementation of a proficient 3D reconstruction process for surgical scenes and instruments, using depth estimation from the Multi-Task Learning model. • Incorporation of a Adversarial Weight Update within our methodology, strategically employed to achieve Pareto optimality for each designated task. 2 Related Work The use of object detection, segmentation, tracking, and depth estimation in medical imaging spans various surgical and diagnostic specialties. Diverse methodologies leverage datasets like the EndoVis, with approaches including single-task learning (STL). Furthermore, 3D reconstruction [ 17 ] plays a crucial role in surgical understanding. Our focus lies on multitask learning for surgical instrument tasks, recognizing its benefits in generalization and computational efficiency. 2.1 Segmentation and object detection CNN models are utilized for binary, semantic, and instance segmentation in surgical tool tracking. TernausNet [ 8 ] , excels in binary segmentation, while ISINet [ 25 ] , TraSeTR [ 25 ] , and AP-MTL [ 9 ] focus on instance segmentation. Considering the computational demands, lighter techniques like binary segmentation can enhance processing speed. Moreover, combining binary segmentation with object detection can yield comparable outcomes to instance and semantic segmentation methods. Advancements in instrument detection include TernausNet-16 [ 8 ] , a CNN extension of U-Net, and methods by Kurmann et al. [ 14 ] for 2D vision-based recognition and Sarikaya et al. [ 20 ] for region proposal network (RPN) and multi-modal two-stream convolutional network. S3Net [ 1 ] employs instance segmentation for detection, but integrating binary segmentation with detection readings offers a more cost-effective approach to achieve instance segmentation. 2.2 Depth Estimation Traditionally hampered by data scarcity, stereo-based 3D reconstruction relied on handcrafted methods [ 21 ] , [ 26 ] validated on specialized datasets [ 5 ] . However, new advancements in monocular depth estimation now rival the capabilities of stereo [ 17 ] and LiDAR setups. This ensures cost-effectiveness and system flexibility, while harnessing the latest developments to provide precise visual guidance in robotic surgery. MTL leverages semantic segmentation for scene context in monocular depth estimation, addressing issues like blurry object boundaries and enhancing accuracy, as pioneered in [ 6 ] . MTL networks optimize efficiency by sharing a single feature extraction backbone, cutting down on inference time and memory usage, enabling models like SENSE [ 11 ] to predict multiple outputs concurrently. 2.3 Multi-Task Learning In computer vision, various MTL models like MaskRCNN [ 7 ] have been devised for concurrent semantic segmentation and object detection tasks. Recently, efforts have been made by UberNet [ 12 ] and AP-MTL [ 9 ] to refine MTL models through multi-phase training approaches and fine-tuning. However, while MTL has seen significant progress in segmentation and detection tasks, it trails behind in depth estimation for comprehensive 3D reconstruction. MSDESIS [ 17 ] attempted multitask depth estimation and segmentation utilizing a stereo camera setup, which could present challenges in hardware and space-limited MIS procedures. Therefore, additional research is required to explore monocular multi-task learning approaches in robotic MIS environments. 3 Methodology The proposed model architecture comprises an Encoder, Decoder, and task-specific heads, integrated into a unified Multi-Task Learning Transformer (MT3DNet) framework is depicted in Fig. 1 . Figure 1 : The proposed Multi-Task Learning Transformer (MT3DNet) architecture. Encoder. The encoder architecture is designed to extract hierarchical features from input data by progressively transforming and downsampling it through multiple stages. Each stage in the encoder consists of a series of operations that refine the representation, leveraging both convolutional and attention-based mechanisms. The stages are stacked sequentially, each having a defined number of layers and different configurations for patch size, overlap, and dimensionality reduction. At the core of the encoder, each stage performs patch merging, where the spatial resolution of the input is reduced while preserving essential features. This is achieved using the concept of overlap patch merging, where overlapping patches are combined to enhance the information retention during downsampling. The input is then passed through a series of encoder blocks, each block consisting of self-attention and feed-forward layers. These blocks are meant to capture long-range dependencies