TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning.
Pérez-García F, Sparks R, Ourselin S. · 2021
Read TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. by Pérez-García F, Sparks R, Ourselin S. (2021) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc8542803.txt.
Introduction
Pérez-García and co-authors present TorchIO, a Python library for medical images in deep learning. Medical images pose special load and prep problems. They come in 2D, 3D, and 4D forms, from X-ray to MRI. Labels are scarce and data sets are heavy.
TorchIO handles loading, preprocessing, augmentation, and patch-based sampling in one toolkit. Its modular design follows the style of PyTorch. It stresses work that others can repeat and trace. That stance backs open science in the field. The library covers histopathology, CT, ultrasound, and diffusion MRI as well. It plugs into related frameworks for image work. The authors aim to ease a common bottleneck.
Catalog record
| CREATOR | Pérez-García F, Sparks R, Ourselin S. |
|---|---|
| YEAR | 2021 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning By Fernando Pérez-García, Rachel Sparks, Sébastien Ourselin Published in: Computer Methods and Programs in Biomedicine DOI: 10.1016/j.cmpb.2021.106236 ABSTRACT Highlights •Open-source Python library for preprocessing, augmentation and sampling of medical images for deep learning.•Support for 2D, 3D and 4D images such as X-ray, histopathology, CT, ultrasound and diffusion MRI.•Modular design inspired by the deep learning framework PyTorch.•Focus on reproducibility and traceability to encourage open-science practices.•Compatible with related frameworks for medical image processing with deep learning. INTRODUCTION Recently, deep learning has become a ubiquitous research approach for solving image understanding and analysis problems. Convolutional neural networks (CNNs) have become the state of the art for many medical imaging tasks including segmentation [1], classification [2], reconstruction [3] and registration [4]. Many of the network architectures and techniques have been adopted from computer vision. Compared to 2D red-green-blue (RGB) images typically used in computer vision, processing of medical images such as MRI, ultrasound (US) or CT presents different challenges. These include a lack of labels for large datasets, high computational costs (as the data is typically volumetric), and the use of metadata to describe the physical size and position of voxels. Open-source frameworks for training CNNs with medical images have been built on top of TensorFlow [5], [6], [7]. Recently, the popularity of PyTorch [8] has increased among researchers due to its improved usability compared to TensorFlow [9], driving the need for open-source tools compatible with PyTorch. To reduce duplication of effort among research groups, improve experimental reproducibility and encourage open-science practices, we have developed TorchIO: an open-source Python library for efficient loading, preprocessing, augmentation, and patch-based sampling of medical images designed to be integrated into deep learning workflows. TorchIO is a compact and modular library that can be seamlessly used alongside higher-level deep learning frameworks for medical imaging, such as the Medical Open Network for AI (MONAI). It removes the need for researchers to code their own preprocessing pipelines from scratch, which might be error-prone due to the complexity of medical image representations. Instead, it allows researchers to focus on their experiments, supporting experiment reproducibility and traceability of their work, and standardization of the methods used to process medical images for deep learning. Motivation The nature of medical images makes it difficult to rely on a typical computer-vision pipeline for neural network training. In Section 1.1.1, we describe challenges related to medical images that need to be overcome when designing deep learning workflows. In Section 1.1.2, we justify the choice of PyTorch as the main deep learning framework dependency of TorchIO. Challenges in medical image processing for deep learning 1.1.1 In practice, multiple challenges must be addressed when developing deep learning algorithms for medical images: 1) handling metadata related to physical position and size, 2) lack of large labeled datasets, 3) high computational costs due to data multidimensionality and 4) lack of consensus for best normalization practices. These challenges are very common in medical imaging and require certain features that may not be implemented in more general-purpose image processing frameworks such as Albumentations [10] or TorchVision [8]. Metadata In computer vision, picture elements, or pixels, which are assumed to be square, have a spatial relationship that comprises proximity and depth according to both the arrangement of objects in the scene and camera placement. In comparison, medical images are reconstructed such that the location of volume elements, or cuboid-shaped voxels, encodes a meaningful 3D spatial relationship. In simple terms, for 2D natural images, pixel vicinity does not necessarily indicate spatial correspondence, while for medical images spatial correspondence between nearby voxels can often be assumed. Metadata, which encodes the physical size, spacing, and orientation of voxels, determines spatial relationships between voxels [11]. This information can provide meaningful context when performing medical image processing, and is often implicitly or explicitly used in medical imaging software. Furthermore, metadata is often used to determine correspondence between images as well as voxels within an image. For example, registration algorithms for medical images typically work with physical coordinates rather than voxel indices. Fig. 1 shows the superposition of an MRI and a corresponding brain parcellation [12] with the same size (181×181) but different origin, spacing and orientation. A native user would assume that, given that the superimposition looks correct and both images have the same size, they are ready for training. However, the visualization is correct only because 3D Slicer [13], the software used for visualization, is aware of the spatial metadata of the images. As CNNs generally do not take spatial metadata into account, training using these images without preprocessing would lead to poor results.Fig. 1Demonstration of the importance of spatial metadata in medical image processing. The size of both the MRI and the segmentation is 181×181. When spatial metadata is taken into account (a), images are correctly superimposed (only the borders of each region are shown for clarity purposes). Images are incorrectly superimposed if (b) origin, (c) orientation or (d) spacing are ignored.Fig. 1 Medical images are typically stored in specialized formats such as Data Imaging and Communications in Medicine (DICOM) or Neuroimaging Informatics Technology Initiative (NIfTI) [11], and commonly read and processed by medical imaging frameworks such as SimpleITK [14] or NiBabel [15]. Limited training data. Deep learning methods typically require large amounts of annotated data, which are often scarce in clinical scenarios due to concerns over patient privacy, the financial and time burden associated with collecting data as part of a clinical trial, and the need for annotations from highly-trained and experienced raters. Data augmentation techniques can be used to increase the size of the training dataset artificially by applying different transformations to each training instance while preserving the relationship to annotations. Data augmentation performed in computer vision typically aims to simulate variations in camera properties, field of view (FOV), or perspective. Traditional data augmentation operations applied in computer vision include geometrical transforms such as random rotation or zoom, color-space transforms such as random channel swapping or kernel filtering such as random Gaussian blurring. Data augmentation is usually performed on the fly, i.e., every time an image is loaded from disk during training. Several computer vision libraries supporting data augmentation have appeared recently, such as Albumentations [10], or imgaug [16]. PyTorch also includes some computer vision transforms, mostly implemented as Pillow wrappers [17]. However, none of these libraries support reading or transformations for 3D images. Furthermore, medical images are almost always grayscale, therefore color-space transforms are not applicable. Additionally, cropping and scaling are more challenging to apply to medical images without affecting the spatial relationships of the data. Metadata should usually be considered when applying these transformations to medical images. In medical imaging, the purpose of data augmentation is designed to simulate anatomical variations and scanner artifacts. Anatomical variation and sample position can be simulated using spatial transforms such as elastic deformation, lateral flipping, or affine transformations. Some artifacts are unique to specific medical image modalities. For example, ghosting artifacts will be present in MRI if the patient moves during acquisition, and metallic implants often produce streak artifacts in CT. Simulation of these artifacts can be useful when performing augmentation on medical images. Computational costs. The number of