Deep histological synthesis from mass spectrometry imaging for multimodal registration

Kimberley M. Bird, Xujiong Ye, Alan M. Race, James M. Brown · 2025

Read Deep histological synthesis from mass spectrometry imaging for multimodal registration by Kimberley M. Bird, Xujiong Ye, Alan M. Race, James M. Brown (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05441.txt.

Introduction

Kimberley Bird and three co-authors show in 2025 how to fake a microscope image from a chemical scan so two tissue imaging methods can be aligned. Histology pictures tissue structure, while mass spectrometry imaging maps its chemistry, and merging them pinpoints where molecules sit in anatomy. The two image types form so differently that direct alignment is a standing problem.

A pix2pix network learns to synthesize a histology-style image from the chemical scan. Aligning the synthetic image with the real one becomes a same-type problem, which standard tools handle well. Alignment quality beat a U-Net baseline on both information and similarity scores. Chemical maps locked to anatomy can sharpen cancer margin studies and drug distribution maps alike.

Catalog record

CREATORKimberley M. Bird, Xujiong Ye, Alan M. Race, James M. Brown
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE6 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05441

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deep histological synthesis from mass spectrometry imaging for multimodal registration
Kimberley M. Bird, Xujiong Ye, Alan M. Race, James M. Brown

Abstract
Registration of histological and mass spectrometry imaging (MSI) allows for more precise identification of structural changes and chemical interactions in tissue. With histology and MSI having entirely different image formation processes and dimensionalities, registration of the two modalities remains an ongoing challenge. This work proposes a solution that synthesises histological images from MSI, using a pix2pix model, to effectively enable unimodal registration. Preliminary results show promising synthetic histology images with limited artifacts, achieving increases in mutual information (MI) and structural similarity index measures (SSIM) of + 0.924 +0.924 and + 0.419 +0.419 , respectively, compared to a baseline U-Net model. Our source code is available on GitHub: https://github.com/kimberley/MIUA2025 .



1
Introduction

Mass spectrometry imaging (MSI) can visualise and localise thousands of known and unknown biomarkers by their molecular mass at a cellular level. Additionally, histological images, such as hematoxylin and eosin (H&E) stained tissues, reveal valuable structural information regarding cell morphology and tissue types. Spatial registration of these modalities allows for precise localisation of biomolecules within structures of interest. However, due to the destructive nature of MSI and H&E, images are typically adjacent tissue slices, increasing the risk of non-homologous information between two sections
[
1
]
. Furthermore, registering thousands of MSI channels to a regular 3-channel histology image is intractable, leading researchers to reduce MSI dimensionality using conventional techniques, such as principal component analysis (PCA)
[
2
]
, t-distributed stochastic neighbor embedding (t-SNE)
[
3
]
, and uniform manifold approximation and projection (UMAP)
[
4
]
.

Recent studies have proposed the use of generative models within digital pathology for a variety of purposes: converting H&E to multiplex immunohistochemistry (IHC)
[
5
]
, producing synthetic IHC from masks for downstream segmentation
[
6
]
, synthetic multi-staining of microscopy tissue
[
7
]
, and generating IHC from H&E
[
8
]
. This paper proposes the use of pix2pix
[
9
]
to generate synthetic histology from MSI to enable downstream unimodal registration. The pix2pix model was compared to a U-Net
[
10
]
model trained on histology, and evaluated based on their perceived quality and similarity to real histology. To our knowledge, this is the first work to perform image-to-image translation in the generation of paired histological and MSI images.

2
Materials and Methods

We used a publicly available dataset of breast tissue desorption electrospray ionization MSI (DESI-MSI) and histology images
[
11
]
. The MSI data was acquired at a spatial resolution of
100\mu m
, and the histology samples were stained using H&E, totalling 111 pairs. The MSI data was pre-processed by applying interpolation rebinning
[
12
]
to standardise m/z values across the entire dataset. After rebinning, all spectra were summed and the top 50 peaks were selected. Subsequently, image pairs were rescaled and resized (using black or white padding for histology where necessary) to match dimensions, followed by affine control point registration. For registration, a three-channel representation of the MSI was used (via either PCA, t-SNE, or UMAP) depending on its visual clarity. For the U-Net experiment, image pairs were split into
32\times 32
patches. The dataset was split into 70% training, 10% validation, and 20% testing.

The U-Net model was developed using Python 3.8.10 and implemented in Keras 2.11.0, using an Intel Core i5-6500 CPU and NVIDIA RTX 2080Ti 11GB GPU. The model had minimal modifications from
[
10
]
(batch size=64, learning rate=0.01, optimiser=Adam, activation=sigmoid, loss=mean squared error). The pix2pix model was implemented in PyTorch 2.4.1, changing several hyperparameters: input image size=
512\times 512
, learning rate=0.00002, and L1 penalty=200. All models were trained until they converged. Performance was evaluated using mutual information (MI) and structural similarity index measure (SSIM).

3
Results and Discussion

Overall, the pix2pix model outperformed the U-Net (which produced unsatisfactory results in off-tissue regions) in terms of MI and SSIM, as shown in Table
1
, with white padding achieving superior MI and very similar SSIM. Fig.
1
shows two examples of synthetic (fake) histology images from the validation set, alongside their corresponding MSI and real histology images. The MSI images shown have a
m/z=885.55213
. Notably, Samples 1 (Fig.
1
A-C) and 2 (Fig.
1
D-F) show definite similarities in the overall tissue shape between real and fake. Furthermore, Sample 2’s real histology (Fig.
1
E) contains blue pen marks in the bottom-left and bottom-right corners, which have not been translated to the fake histology (Fig.
1
F). Additionally, both samples show slightly darker colouration in correct parts.

Table 1:
Comparison of experimental results for synthesising histological images using a U-Net and pix2pix models, where
B
denotes black padding and
W
white padding.

MI

SSIM

U-Net (
B
)

0.429

0.530

pix2pix (
B
)

1.349

0.949

pix2pix (
W
)

1.353

0.949

Figure 1:
Two examples of synthesised histology tissue samples (C+F) compared to their corresponding MSI (A+D) and real histology (B+E).

4
Conclusion

Our results show that image-to-image translation can synthesise histology images from MSI, which are qualitatively and quantitatively similar to real examples, serving as a promising solution for registration of the two modalities. The proposed pipeline utilises a pix2pix model which synthesises histological images using peak-picked MSI data, resulting in increases in MI and SSIM of
+0.924
and
+0.419
, respectively, compared to a U-Net model. Visually, the results resemble their real counterpart’s tissue shape and have minimal artifacts. Future work will focus on increasing the training dataset size and diversity, and comparing against state-of-the-art image-to-image translation models (e.g. Palette
[
13
]
).

Read the complete text