Sparse Autoencoders for Interpretable Medical Image Representation Learning
Philipp Wesp, Robbie Holland, Vasiliki Sideri-Lampretsa, Sergios Gatidis · 2026
Read Sparse Autoencoders for Interpretable Medical Image Representation Learning by Philipp Wesp, Robbie Holland, Vasiliki Sideri-Lampretsa, Sergios Gatidis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23794.txt.
Introduction
Wesp and co-authors use sparse autoencoders to make medical image features easier to read. Vision foundation models encode scans in dense vectors that clinicians cannot inspect. The team trains autoencoders on embeddings from BiomedParse and DINOv3, using 909,873 CT and MRI slices. The sparse features rebuild the vectors well and keep most task performance with just 10 features. They also match named concepts and support search by plain language. The authors see a path to concept-based medical vision.
Catalog record
| CREATOR | Philipp Wesp, Robbie Holland, Vasiliki Sideri-Lampretsa, Sergios Gatidis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18202 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23794
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sparse Autoencoders for Interpretable Medical Image Representation Learning Philipp Wesp, Robbie Holland, Vasiliki Sideri-Lampretsa, Sergios Gatidis Abstract Vision foundation models (FMs) achieve state-of-the-art performance in medical imaging. However, they encode information in abstract latent representations that clinicians cannot interrogate or verify. The goal of this study is to investigate Sparse Autoencoders (SAEs) for replacing opaque FM image representations with human-interpretable, sparse features. We train SAEs on embeddings from BiomedParse (biomedical) and DINOv3 (general-purpose) using 909,873 CT and MRI 2D image slices from the TotalSegmentator dataset. We find that learned sparse features: (a) reconstruct original embeddings with high fidelity (R 2 up to 0.941) and recover up to 87.8 % of downstream performance using only 10 features (99.4 % dimensionality reduction), (b) preserve semantic fidelity in image retrieval tasks, (c) correspond to specific concepts that can be expressed in language using large language model (LLM)-based auto-interpretation. (d) bridge clinical language and abstract latent representations in zero-shot language-driven image retrieval. Our work indicates SAEs are a promising pathway towards interpretable, concept-driven medical vision systems. Code repository: https://github.com/pwesp/sail . 1 Introduction Figure 1: (A) A Sparse Autoencoder replaces opaque dense FM embeddings with a sparse feature space. (B) Sparse fingerprint retrieval matches images by cosine similarity over k top-activated features. (C) A VLM generates a concept description for each feature from its top-activating images and metadata. (D) An LLM maps a clinical text query to matching feature concepts for zero-shot image retrieval. Vision foundation models (FMs) achieve strong performance in medical imaging tasks such as segmentation, classification, and retrieval, but encode information in abstract, low-dimensional feature representations [ 13 , 14 ] . At the same time, clinical deployment demands interpretability: physicians must justify decisions, detect failure modes, and document reasoning, yet model internals remain inaccessible [ 12 ] . This creates a fundamental misalignment between abstract learned representations and the anatomical and clinical concepts that clinicians reason with. Mechanistic interpretability aims to reverse-engineer model internals into human-understandable components. Sparse Autoencoders (SAEs) [ 4 , 7 ] are a leading approach, decomposing polysemantic activations in large language models (LLMs) into monosemantic features [ 8 ] that each correspond to a single coherent concept. A recent study applied SAEs to chest radiograph embeddings, demonstrating that a small number of interpretable sparse features can represent clinically relevant visual concepts and support radiology report generation [ 1 ] . That study, however, was restricted to a single modality and a single FM architecture with paired text supervision for concept labelling, leaving open whether anatomical structure emerges in self-supervised models across CT, MRI, and diverse anatomical regions. This raises a central question: do anatomical and clinical concepts emerge from self-supervised medical vision training without explicit labels, and can SAEs expose this structure consistently across architecturally distinct foundation models? To this end, we train Matryoshka SAEs [ 6 ] with BatchTopK sparsification [ 5 ] on frozen embeddings from BiomedParse [ 19 ] (biomedical FM) and DINOv3 [ 17 ] (general-purpose FM), alongside a random-weight baseline to isolate learned representational structure from architectural effects, across 909,873 CT and MRI images from the TotalSegmentator dataset [ 18 , 2 ] (Fig. 1 ). We find that sparse features (a) faithfully reconstruct dense embeddings (R 2 up to 0.941) and recover 87.8 % of downstream performance with only 10 features, (b) preserve 97.7 % of dense retrieval quality with five-feature fingerprints, (c) correspond to monosemantic concepts verified by an independent large language model judge, and (d) enable zero-shot language-driven image retrieval bridging clinical text and medical image content. These findings indicate that self-supervised vision FMs implicitly encode anatomy-aligned structure that SAEs can expose as language-describable sparse features, a step toward interpretable medical AI aligned with human language. 2 Methods We train SAEs (the only optimised parameters) on frozen, precomputed embeddings from three vision FMs: BiomedParse [ 19 ] (1536-dim, biomedical FM), DINOv3 [ 17 ] (1024-dim, general-purpose self-supervised ViT), and an untrained BiomedParse model with randomly initialised weights (1536-dim, random-weight baseline) to isolate learned representational structure from architectural effects. 2.1 Sparse Autoencoder We adopt the Matryoshka SAE architecture [ 6 ] with L=4 nested dictionary levels of increasing size [D_{1},D_{2},D_{3},D_{4}] . A single shared linear encoder projects the input into D_{4} pre-activation codes. Level \ell uses only the first D_{\ell} codes as a prefix subset, so that early levels capture coarse structure and later levels refine it progressively. A single shared decoder (encoder weights transposed, columns normalised to unit norm) reconstructs the input at each level by padding smaller-level activations with zeros. During training, we apply BatchTopK sparsification [ 5 ] : k features are active per sample on average across the batch, allowing flexible per-sample allocation unlike fixed per-sample TopK. At inference, a JumpReLU [ 15 ] threshold, estimated as a running average of the minimum kept activation during training, replaces BatchTopK. The training objective is the mean squared error (MSE) between input and reconstruction, averaged across all L levels, with no auxiliary sparsity or diversity penalties. Monosemanticity scoring. To compare configurations, we score each feature as M(f)=C(f)\times S(f) , where coherence C(f) is the null-adjusted mean pairwise Jaccard similarity over organ sets of its top-10 activating samples and specificity S(f) is the normalized inverse entropy over the organ label distribution. The configuration-level score M_{\mathrm{config}} is the mean M(f) of the top-10 features per configuration. 2.2 Interpretability Evaluation We evaluate the interpretability of learned sparse features through three complementary demonstrations. Sparse Fingerprint Retrieval. We define a sparse fingerprint as the k most activated features and their values per image, and retrieve similar images by cosine similarity over fingerprints. Retrieval quality is measured as mean cosine similarity to the reference in the dense embedding space, with dense retrieval as the upper bound. Automated Feature Interpretation. To assess whether individual features encode interpretable and consistent concepts, we greedily select the 5 most dissimilar samples from the top-20 activating images for the top-250 most monosemantic ( M score) features using cosine similarity. We then prompt the vision language model (VLM) MedGemma 27B [ 16 ] to generate a natural-language concept description from their images and metadata (modality, orientation, anatomy, demographics) [ 9 , 3 ] . A VLM judge (separate MedGemma 27B) then receives the same images and five candidate descriptions, one true and four drawn from other features, and must identify the correct one. The rank of the true concept ( 1=\text{best} , 5=\text{worst} ) quantifies interpretability. Language-Driven Image Retrieval. An LLM identifies feature descriptions that match a clinical text query, assembling a sparse fingerprint from their mean activations for cosine retrieval without a reference image. This zero-shot procedure demonstrates that sparse feature concepts can bridge human language and medical image content. 3 Experiments & Results Figure 2: SAE quality and performance recovery across 96 configurations per FM (DINOv3: blue, BiomedParse: orange, random baseline: grey). (A–D) Reconstruction fidelity (R 2 ), downstream ROC-AUC, alive features, and monosemanticity score vs. L0 sparsity. (E–G) Performance recovery using only the top- N features ( N=1,3,10,50 ). We evaluate Matryoshka SAEs on BiomedParse and DINOv3 embeddings from the TotalSegmentator dataset: 1,844 scans (1,228 CT, 616 MRI) from 10 institutions, yielding 909,873 2D images with 138 per-image metadata fields spanning anatomy presence, imaging parameters and demographics.