Controllable Skin Synthesis via Lesion-Focused Vector Autoregression Model
Jiajun Sun, Zhen Yu, Siyuan Yan, Jason J. Ong, Zongyuan Ge, Lei Zhang · 2025
Read Controllable Skin Synthesis via Lesion-Focused Vector Autoregression Model by Jiajun Sun, Zhen Yu, Siyuan Yan, Jason J. Ong, Zongyuan Ge, Lei Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-19626.txt.
Introduction
Jiajun Sun and five co-authors presented LF-VAR in twenty twenty-five, a model that generates synthetic skin images with control over lesions. Real clinical skin photos are scarce, which starves training data for diagnostic AI. Earlier generators made poor images and could not place a lesion of a chosen type where the user wanted.
Their system first encodes images into discrete tokens with a lesion-focused autoencoder, then builds new images token by token with a transformer. Measured lesion scores and type labels enter as conditions, and language prompts can steer the result. Across seven lesion types it earned the best image quality scores, improving on the prior best by about six percent. The aim is plentiful, realistic training images that respect clinical detail.
Catalog record
| CREATOR | Jiajun Sun, Zhen Yu, Siyuan Yan, Jason J. Ong, Zongyuan Ge, Lei Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-19626
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Controllable Skin Synthesis via Lesion-Focused Vector Autoregression Model Jiajun Sun, Zhen Yu, Siyuan Yan, Jason J. Ong, Zongyuan Ge, Lei Zhang Abstract Skin images from real-world clinical practice are often limited, resulting in a shortage of training data for deep-learning models. While many studies have explored skin image synthesis, existing methods often generate low-quality images and lack control over the lesion’s location and type. To address these limitations, we present LF-VAR , a model leveraging quantified lesion measurement scores and lesion type labels to guide the clinically relevant and controllable synthesis of skin images. It enables controlled skin synthesis with specific lesion characteristics based on language prompts. We train a multiscale lesion-focused Vector Quantised Variational Auto-Encoder (VQVAE) to encode images into discrete latent representations for structured tokenization. Then, a Visual AutoRegressive (VAR) Transformer trained on tokenized representations facilitates image synthesis. Lesion measurement from the lesion region and types as conditional embeddings are integrated to enhance synthesis fidelity. Our method achieves the best overall FID score (average 0.74) among seven lesion types, improving upon the previous state-of-the-art (SOTA) by 6.3% . The study highlights our controllable skin synthesis model’s effectiveness in generating high-fidelity, clinically relevant synthetic skin images. Our framework code is available at https://github.com/echosun1996/LF-VAR. 1 Introduction High-quality skin images are essential for training and validating medical artificial intelligence (AI) models. However, skin images from real-world clinical practice often suffer from scarcity and long-tail imbalances, leading to biased outcomes or algorithmic unfairness [ 24 ] . Furthermore, privacy concerns, legal restrictions, and financial burdens pose substantial challenges in acquiring diverse, high-quality skin images [ 9 , 8 ] . Generative models have emerged as a potential solution to address these data-related challenges. For instance, SkinDiff [ 13 ] employs a latent diffusion framework to generate labelled skin lesion images, while Derm-T2IM [ 7 ] leverages diffusion models with lesion-specific textual prompts to generate dermoscopic images. However, diffusion models are limited by their dependence on large-scale training data and computationally intensive synthesis processes [ 22 , 3 ] , making them less feasible for resource-constrained clinical settings. Recent advances in autoregressive models, particularly Vector AutoRegressive (VAR) Transformers, have demonstrated competitive performance in general image synthesis tasks while requiring fewer computational resources and less training data than diffusion models [ 25 ] . However, our evaluation of the VAR model in skin image synthesis revealed two critical limitations: First, existing implementations lack mechanisms to guide synthesis using diagnostically relevant features, resulting in generated images with limited pathological diversity and fidelity. Second, without explicit separation of lesions and normal skin regions, VAR-generated images exhibit unnatural transitions and background noise, reducing their realism. To address these challenges, we propose LF-VAR , a controllable synthesis model that leverages masks and lesion type as prompts to enable precise and flexible skin lesion synthesis. Additionally, LF-VAR integrates quantified lesion measurements—such as shape and texture—to guide our model over clinically relevant features, ensuring high-fidelity synthetic images with well-preserved pathological characteristics. The primary contributions of our work are: 1. We propose a lightweight encoder that transforms quantified lesion measurements into discriminative latent representations, which enables precise control over clinically relevant features and allows high-fidelity synthesis. 2. We implement a unified model that incorporates segmentation masks and lesion type, enabling controlled synthesis of lesions across different lesion categories. This design allows for the flexible generation of diverse pathological presentations while maintaining consistency with clinical descriptors. 3. We integrate pixel-level segmentation into the generative pipeline to focus lesion synthesis while minimizing background region artifacts. It enables high-quality image synthesis by maintaining a clear distinction between lesions and normal skin regions. 4. We conduct extensive validation indicating robust synthesis across intra-class, inter-class, and cross-dataset scenarios. Our evaluation confirms the model’s effect in high-fidelity clinically relevant skin synthetic. Figure 1: The framework of LF-VAR . A structured VQ-VAE encoded images into K token maps, shared with the VAR, and was trained with lesion-focused loss. Lesion measurement scores were extracted from lesion areas and encoded as \mathbf{F_{q}} , combined with lesion type embedding \mathbf{S} , and incorporated into the VAR as a conditional token to enable controlled intra-class skin lesion synthesis. Inter-class synthesis was achieved through the VAR model by querying the lesion type average measurements codebook. 2 Method The proposed synthesis model, as shown in Fig. 1 consisted of two distinct stages: (1) a multiscale lesion-focused Vector Quantised Variational Auto-Encoder (VQ-VAE) [ 19 ] , which was employed to acquire compact discrete image representations, and (2) a VAR [ 25 ] model trained on tokenised representations to facilitate image synthesis. In the second stage, measurement scores extracted from the skin lesion region were incorporated into the transformer, along with lesion types as conditional embeddings, to improve the fidelity of the generated images. 2.1 Multi-Scale Lesion-Focused VQ-VAE A multi-scale lesion-focused VQ-VAE was trained to encode images into discrete latent representations, enabling structured and hierarchical tokenization. Given an input image \mathbf{I}\in\mathbb{R}^{H\times W\times C} , the encoder E(\cdot) extracted feature maps, yielding a latent feature map f=E(\mathbf{I}) , where f\in\mathbb{R}^{h\times w\times d} . A multi-scale quantization process was subsequently applied to discretize the latent representation: \footnotesize\mathbf{R}=Q(f)=\{R_{k}\}_{k=1}^{K},\hskip 8.50012ptR_{k}\in\mathbb{Z}^{h_{k}\times w_{k}}, (1) where Q(\cdot) denoted the quantization function, K represented the number of scales, and each R_{k} was a discrete token grid at scale k . The decoder networks D(\cdot) and \hat{D}(\cdot) were responsible for reconstructing the feature map and the final image, respectively, where the reconstructed feature map was given by \hat{f}=D(\mathbf{R}) and the final reconstructed image by \hat{\mathbf{I}}=\hat{D}(\hat{f}) . To enhance reconstruction fidelity, a loss function was introduced to enforce both pixel-level and perceptual constraints while ensuring feature consistency in non-masked regions across different scales. Overall loss function was defined as: \footnotesize\mathcal{L}=\|\mathbf{I}-\hat{\mathbf{I}}\|_{2}+\sum_{k=1}^{K}\|(1-M_{k})\odot(R_{k}-\hat{R}_{k})\|_{2}+\|f-\hat{f}\|_{2}+\lambda_{P}\mathcal{L}_{P}(\hat{\mathbf{I}})+\lambda_{G}\mathcal{L}_{G}(\hat{\mathbf{I}}), (2) where M_{k}\in\{0,1\}^{h_{k}\times w_{k}} was the lesion mask at scale k . The second term was lesion-focused loss, enforcing similarity between real and reconstructed feature tokens in non-masked areas across all scales. \mathcal{L}_{P} represented a perceptual loss, while \mathcal{L}_{G} represented a discriminative loss. \lambda_{P} and \lambda_{G} were the loss weights. 2.2 Lesion Measurement Embedded VAR Transformer In clinical dermatology, quantified lesion measurements—structured scores that captured morphological characteristics such as lesion texture and shape—were essential for both diagnosis and prognosis [ 24 , 16 , 15 ] . These structured scores, derived from mask regions rather than full images, offered robust feature representations. Despite their clinical importance, the potential of these structured measurements to guide medical image synthesis remained largely unexplored. To bridge this gap, we incorporated lesion measurement scores as conditioning signals for controlled skin lesion synthesis. Specifically, during intra-class synthesis, lesion measurement scores \varepsilon_{\text{r}} were extracted from the original images using PyRadiomics [ 10 ] formulated as \varepsilon_{\text{r}}=E_{\text{ext}}(\mathbf{I,M}) . These measurement scores included shape, histogram, texture, and clinical attributes. To enable autoregressive modelling of discrete tokens, a VAR Transformer was employed, conditioning synthesis on learned embeddings. Along with extracted multi-scale tokens, measurement scores were incorporated via an encoding function, \mathbf{F}_{\text{q}}=E_{\text{q}}(\varepsilon_{\text{r}}) . The measurement scores \varepsilon_{\text{r}} were transformed via a linear projection, followed by layer normalization and a SiLU activation function. For inter-class synthesis, an additional codebook of class-average measurements was maintained. During the intra-class synthesis, \varepsilon_{\text{r}} was used to update the corresponding skin lesion entry in the codebook by computing the average measurements. During the