The Learnability Gap in Medical Latent Diffusion
Mischa Dombrowski, Felix Nützel, Bernhard Kainz · 2026
Read The Learnability Gap in Medical Latent Diffusion by Mischa Dombrowski, Felix Nützel, Bernhard Kainz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17087.txt.
Introduction
Dombrowski's team identifies the learnability gap in medical diffusion: large pretrained autoencoders encode medical classification features, yet their latent codes resist learning. Across five autoencoder families and four tests spanning chest, skin, CT, and heart imaging, the gap persists regardless of architecture or tuning, and fine-tuning does not close it. Noise-conditioned latent classifiers with FiLM layers and distillation narrow the gap while giving 64-fold throughput and 120-fold memory gains. The analysis frames latent-space structure, not fidelity or specificity, as the main block to closing that gap.
Catalog record
| CREATOR | Mischa Dombrowski, Felix Nützel, Bernhard Kainz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26138 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17087
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Learnability Gap in Medical Latent Diffusion Mischa Dombrowski, Felix Nützel, Bernhard Kainz Abstract Generative data augmentation with latent diffusion models is a promising strategy for addressing class imbalance in medical imaging, yet current approaches focus on perceptual fidelity and domain-specific autoencoder fine-tuning while neglecting a more fundamental bottleneck. We identify and formalize the learnability gap : large-scale pretrained autoencoders faithfully encode discriminative features for medical classification, as evidenced by near-lossless performance in reconstruction space, yet their latent representations are structured in ways that are difficult for classifiers to learn from. Across five autoencoder families and four medical benchmarks spanning chest radiography, dermatoscopy, computed tomography, and echocardiography, we show that this gap persists regardless of architecture, initialization strategy, or hyperparameter tuning, and that medical-domain fine-tuning of the autoencoder does not close it. To probe and partially narrow the gap, we develop noise-conditioned latent classifiers with FiLM layers and image-space distillation that offer 64 × 64{\times} throughput and 120 × 120{\times} memory gains over image-space models while serving as diagnostic tools for latent space quality. Our analysis provides a new framework for evaluating autoencoder latent spaces and identifies their structure, rather than their fidelity or domain specificity, as the primary obstacle to closing the performance gap between real and synthetic medical training data. 1 Introduction Clinical datasets follow long-tailed distributions almost universally: common diagnoses dominate while rare but critical conditions populate the tail [ 14 , 15 , 22 ] . In chest radiography, “No Finding” can exceed 60% of studies while pneumomediastinum appears in fewer than 0.1% [ 16 ] . Similar imbalances arise in dermatoscopy [ 3 , 35 ] , computed tomography [ 9 ] , and congenital heart disease screening [ 37 ] . Models trained on such distributions systematically underdiagnose rare conditions [ 2 , 17 ] , and acquiring additional labeled examples is expensive and privacy-constrained [ 32 ] . Latent diffusion models [ 33 , 29 , 19 ] offer a compelling path: a hospital trains a generative model locally and shares synthetic samples on demand, a privacy-preserving alternative to federated learning [ 32 , 26 ] . The practical value of this pipeline depends on whether generated images carry the discriminative features downstream classifiers need, particularly for tail classes. Current work addresses this through perceptual fidelity (FID [ 11 ] ) and domain-specific autoencoder fine-tuning [ 36 ] , implicitly treating the latent space as a neutral intermediary. Recent studies on natural images challenge this assumption: latent spaces contain high-frequency artifacts [ 34 ] , lack geometric equivariance [ 20 , 41 ] , and benefit from alignment with pretrained representations [ 39 , 21 , 38 , 40 , 8 ] . The emerging view is that latent space structure actively shapes what a generative model can learn [ 1 , 7 ] , yet the implications for class-conditional generation in long-tailed medical settings remain unexplored. We make this question precise through a systematic study across five autoencoder families and four clinical benchmarks. Reconstruction-space classifiers match image-space performance ( p{=}0.375 , Wilcoxon signed-rank test), confirming that the autoencoders faithfully preserve discriminative information. Yet latent-space classifiers suffer a substantial drop despite extensive architecture search, distillation, and noise-conditional training ( p=9.5\times 10^{-7} , Wilcoxon across all 20 AE \times dataset pairs). We term this the learnability gap (Fig. 1 ): the information is present but structured in a way that resists direct learning. This gap propagates to generative augmentation: if the diffusion model cannot learn class-discriminative structure from the latents, its samples will lack the pathological markers classifiers rely on. Our contributions are: (1) We identify and quantify the learnability gap across five autoencoder families and four medical benchmarks, showing it is systematic ( p=9.5\times 10^{-7} ) and independent of reconstruction fidelity, domain specificity, and initialization strategy. (2) We develop noise-conditioned latent classifiers with FiLM layers and image-space distillation that partially narrow the gap and offer 64{\times} throughput gains, making them practical for rejection sampling and quality filtering in generative pipelines. (3) We provide a controlled analysis isolating the sources of the gap through systematic ablations of fine-tuning, distillation, and noise conditioning, establishing latent space structure as the bottleneck for long-tail medical synthesis. Related work. Long-tailed medical classification has been extensively benchmarked [ 14 , 15 , 22 ] , with LDAM [ 2 ] , decoupled training [ 17 ] , and focal loss [ 23 ] as standard remedies. Generative augmentation via class-balancing diffusion [ 31 ] , guided adapters [ 27 ] , synthetic pretraining [ 26 ] , and diversity control [ 6 ] has shown promise. Latent diffusion [ 33 , 12 , 29 , 19 ] relies on autoencoder latent spaces whose structure significantly impacts generation quality through high-frequency artifacts [ 34 ] , equivariance deficits [ 20 , 41 ] , and spectral biases [ 7 ] . Representation alignment methods [ 39 , 21 , 38 , 40 , 8 ] improve latent spaces using pretrained encoders such as DINOv2 [ 28 ] , and latent-space learnability has been studied for natural images [ 1 ] and memorization detection [ 5 ] . Domain-specific autoencoders preserve clinical features [ 36 ] . Classifier guidance at inference [ 4 , 13 ] and autoguidance [ 18 ] improve sample quality but do not address the underlying representation. We differ by providing the first systematic diagnosis of the learnability gap in medical latent spaces and by developing noise-conditioned latent classifiers as both diagnostic and practical tools for generative pipelines. 2 Method Figure 1: Method overview. A frozen pretrained autoencoder maps images to latent space and back. The learnability gap: reconstruction-space classifiers match image space, but latent-space classifiers degrade substantially. A ConvNeXt-Tiny student classifier with FiLM-based noise conditioning is distilled from an image-space teacher. An overview is given in Fig. 1 . We operate in the latent space of a frozen pretrained autoencoder. Let x_{\text{img}} denote an image, x=E(x_{\text{img}}) its latent encoding, and \tilde{x}_{\text{img}}=G(x) the decoded reconstruction. All latents are channel-wise normalized over the training set. Three evaluation spaces. To isolate the effect of latent encoding from information loss, we define three evaluation spaces: image space (classifiers trained on original images x_{\text{img}} ), latent space (classifiers trained on latent codes x ), and reconstruction space (classifiers trained on decoded reconstructions \tilde{x}_{\text{img}}=G(E(x_{\text{img}})) ). If latent-space classifiers underperform reconstruction-space classifiers, the gap cannot be caused by missing information (since the decoder can recover it) but must instead reflect how the latent space structures that information. This three-way comparison is the core diagnostic tool of our analysis. Noise-conditioned latent classifier. Our core component is a classifier f_{\phi}(x_{\sigma},\sigma) that predicts the class label c from noisy latents at any noise level \sigma , forcing recovery of semantic information even at low signal-to-noise ratios. We adopt a ConvNeXt-Tiny [ 24 ] backbone operating directly on the latent tensor, chosen for its strong performance in prior latent-space studies [ 5 ] . The noise level \sigma is embedded via Fourier features followed by an MLP with SiLU activations, and injected at the end of each ConvNeXt stage through Feature-wise Linear Modulation (FiLM) [ 30 ] : \gamma,\beta=We(\sigma) ; x^{\prime}=x\odot(1+\gamma)+\beta . FiLM parameters are zero-initialized so