Segment Anything for Dendrites from Electron Microscopy
1 st Zewen Zhuo * A.I. Virtanen Institute for Molecular Sciences University of Eastern Finland Kuopio, Finland [email protected] 2 nd Ilya Belevich Electron Microscopy Unit Institute of Biotechnology University of Helsinki Helsinki, Finland [email protected] 3 rd Ville Leinonen Department of · 2024
Read Segment Anything for Dendrites from Electron Microscopy by 1 st Zewen Zhuo * A.I. Virtanen Institute for Molecular Sciences University of Eastern Finland Kuopio, Finland [email protected] 2 nd Ilya Belevich Electron Microscopy Unit Institute of Biotechnology University of Helsinki Helsinki, Finland [email protected] 3 rd Ville Leinonen Department of (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02562.txt.
Description
Abstract Segmentation of cellular structures in electron microscopy (EM) images is fundamental to analyzing the morphology of neurons and glial cells in the healthy and diseased brain tissue. Current neuronal segmentation applications are based on convolutional neural networks (CNNs) and do not effectively capture global relationships within images.
Catalog record
| CREATOR | 1 st Zewen Zhuo * A.I. Virtanen Institute for Molecular Sciences University of Eastern Finland Kuopio, Finland [email protected] 2 nd Ilya Belevich Electron Microscopy Unit Institute of Biotechnology University of Helsinki Helsinki, Finland [email protected] 3 rd Ville Leinonen Department of |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Segment Anything for Dendrites from Electron Microscopy Thanks: * Corresponding Authors 1 st Zewen Zhuo * A.I. Virtanen Institute for Molecular Sciences University of Eastern Finland Kuopio, Finland [email protected] 2 nd Ilya Belevich Electron Microscopy Unit Institute of Biotechnology University of Helsinki Helsinki, Finland [email protected] 3 rd Ville Leinonen Department of Medicine Faculty of Health Sciences University of Eastern Finland Kuopio, Finland [email protected] 4 th Eija Jokitalo Electron Microscopy Unit Institute of Biotechnology University of Helsinki Helsinki, Finland [email protected] 5 th Tarja Malm A.I. Virtanen Institute for Molecular Sciences University of Eastern Finland Kuopio, Finland [email protected] 6 th Alejandra Sierra A.I. Virtanen Institute for Molecular Sciences University of Eastern Finland Kuopio, Finland [email protected] 7 th Jussi Tohka * A.I. Virtanen Institute for Molecular Sciences University of Eastern Finland Kuopio, Finland [email protected] Abstract Segmentation of cellular structures in electron microscopy (EM) images is fundamental to analyzing the morphology of neurons and glial cells in the healthy and diseased brain tissue. Current neuronal segmentation applications are based on convolutional neural networks (CNNs) and do not effectively capture global relationships within images. Here, we present DendriteSAM, a vision foundation model based on Segment Anything, for interactive and automatic segmentation of dendrites in EM images. The model is trained on high-resolution EM data from healthy rat hippocampus and is tested on diseased rat and human data. Our evaluation results demonstrate better mask quality compared to the original and other fine-tuned models, leveraging the features learned during training. This study introduces the first implementation of vision foundation models in dendrite segmentation, paving the path for computer-assisted diagnosis of neuronal anomalies. IEEE Copyright Notice © 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Accepted to be Published in: Proceedings of the 2025 IEEE International Conference on Image Processing Applications and Systems (IPAS 2025), 9-11 January, 2025, Lyon, France. I Introduction Advancements in serial block-face scanning electron microscopy (SBF-SEM) allow researchers to capture brain tissue images at nanometer resolution across hundreds of micrometers [ 1 , 2 ] . Segmenting complex ultrastructures such as axons, soma, and dendrites from electron microscopy (EM) data is critical for analyzing morphological parameters or reconstructing 3-dimensional (3D) neuronal networks. Dendrites, in particular, are intricate and varied, with numerous studies linking dendritic pathology to neurodegenerative diseases such as Alzheimer’s and Parkinson’s [ 3 , 4 , 5 , 6 ] . While manual dendrite annotation has long been the benchmark, it is labor-intensive and time-consuming [ 7 , 8 ] . Deep learning, especially convolutional neural networks (CNNs), has improved segmentation efficiency, but CNNs struggle with generalization across diverse datasets. Inspired by the success of large language models (LLMs), there is growing interest in applying foundation models to computer vision tasks. The name, foundation models, originates from their fundamentally central yet inherently incomplete nature, serving as a robust foundation that can be adapted to downstream tasks [ 9 ] . Models such as GPT-4 [ 10 ] , PaLM-2 [ 11 ] , and LLaMA [ 12 ] have benefited from the sheer scale of data they have been trained on and have demonstrated to generalize beyond the training data. Research on foundation models in computer vision, such as the segment anything model (SAM) [ 13 ] and segment-everything-everywhere model (SEEM) [ 14 ] , demonstrates exceptional versatility in segmentation tasks. Notably, trained on around 1 billion masks from 11 million images, SAM is a state-of-the-art (SOTA) segmentation model. SAM has been widely studied and integrated into various downstream tasks, such as zero-shot abdominal organ segmentation in computed tomography (CT) [ 15 ] and skull-stripping in brain magnetic resonance imaging (MRI) [ 16 ] . SAM’s robustness has been found to be remarkable against various types of corruption, with the exception of blur-related corruption [ 17 ] . In spite of its generalizability gained from large training data, SAM’s performance degrades drastically when handling objects in medical imaging with blurred boundaries or weak contrast [ 18 ] . Therefore, further fine-tuning is essential for the precise segmentation of such objects. Here, we introduce DendriteSAM, a vision foundation model for segmenting dendrites from large-scale EM volumes, and evaluate it internally as well as externally and across subjects and species. Our contributions are: • we present the first application of a vision foundation model specialized in dendrite segmentation; • we evaluate DendriteSAM’s robustness both internally and externally through comprehensive quantitative and qualitative evaluation and demonstrate improved segmentation performance compared to SOTA segmentation models; • we investigate the impact of different image pre-processing methods on dendrite segmentation. II RELATED WORK Proposed in 2023, SAM is a groundbreaking image segmentation model. It offers SOTA performance by exploiting a gigantic scale of training data, enabling it to handle a wide range of segmentation tasks with high accuracy and efficiency. The model incorporates three fundamental modules: an image encoder, a prompt encoder, and a mask decoder. The image encoder relies on vision transformers (ViTs) [ 19 ] and masked autoencoders (MAE) [ 20 ] to translate input images into multi-dimensional embeddings. SAM’s image encoder varies in the number of parameters and is available in three different scales: ViT-Base (ViT-B), ViT-Large (ViT-L), and ViT-Huge (ViT-H). The prompt encoder supports sparse prompts (points, boxes, text) and dense prompts (masks) as input, indicating what to segment in the images. After computing the mask embeddings and prompt embeddings, a lightweight mask decoder is designed to efficiently map the image embeddings, prompt embeddings, and output tokens to masks with confidence scores. The model was evaluated in multiple downstream tasks, such as edge detection, instance segmentation, and object proposal generation through zero-shot learning [ 13 ] . SAM’s utilization on medical images, including zero-shot and few-shot learning, has been broadly investigated. A seminal work in the medical context is MedSAM [ 21 ] , a vision foundation model designed for universal medical image segmentation. MedSAM was trained on 1,570,263 medical image-mask pairs derived from 10 medical imaging modalities, including MRI, CT, X-ray, and ultrasound, among others. This extensive training endowed the model with the ability to segment objects in various types of medical images and anomalies. In addition, it compensated for SAM’s limitation in segmenting medical objects, such as polyps [ 18 ] . The model incorporated the architecture from SAM and initialized the parameters with pretrained SAM using the ViT-B image encoder. To simplify training, only box prompts were considered, and the prompt encoder was frozen in the training phase. This model was evaluated using dice similarity coefficient (DSC) and demonstrated narrower DSC contribution compared to default SAM as well as modality-wise CNN models, revealing MedSAM’s robustness across a variety of tasks. On the other hand, the fixed mask encoder and solitary box prompt hindered the model from iteratively learning and improving from input prompts, compromising the model’s performance to a degree. In addition to universal medical image segmentation, segmentation specialized in microscopy images is a crucial component in histological and anatomical analysis.