Inference-Time Scaling of Diffusion Models for Infrared Data Generation
Kai A. Horstmann, Maxim Clouser, Kia Khezeli · 2025
Read Inference-Time Scaling of Diffusion Models for Infrared Data Generation by Kai A. Horstmann, Maxim Clouser, Kia Khezeli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07362.txt.
Introduction
Kai A Horstmann, Maxim Clouser, and Kia Khezeli generate synthetic infrared images without needing a foundation-scale infrared dataset. Infrared cameras enable temperature-based scene understanding when visible imaging fails, but labeled infrared data is scarce because labeling it takes specialized expertise. Training a big generative model from scratch is therefore out of reach. The authors instead adapt FLUX.1-dev, a leading text-to-image diffusion model, to infrared with parameter-efficient fine-tuning on a small sample.
Quality is then scaled at inference time: a domain-adapted CLIP-based verifier guides the diffusion sampling toward generations that better match the prompt and the infrared domain. On the KAIST multispectral pedestrian benchmark, guided sampling cut the FID score by ten percent against unguided baselines. The authors argue inference-time guidance is a promising way to bridge domain gaps wherever data is limited.
Catalog record
| CREATOR | Kai A. Horstmann, Maxim Clouser, Kia Khezeli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07362
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inference-Time Scaling of Diffusion Models for Infrared Data Generation Kai A. Horstmann, Maxim Clouser, Kia Khezeli Abstract Infrared imagery enables temperature-based scene understanding using passive sensors, particularly under conditions of low visibility where traditional RGB imaging fails. Yet, developing downstream vision models for infrared applications is hindered by the scarcity of high-quality annotated data, due to the specialized expertise required for infrared annotation. While synthetic infrared image generation has the potential to accelerate model development by providing large-scale, diverse training data, training foundation-level generative diffusion models in the infrared domain has remained elusive due to limited datasets. In light of such data constraints, we explore an inference-time scaling approach using a domain-adapted CLIP-based verifier for enhanced infrared image generation quality. We adapt FLUX.1-dev, a state-of-the-art text-to-image diffusion model, to the infrared domain by finetuning it on a small sample of infrared images using parameter-efficient techniques. The trained verifier is then employed during inference to guide the diffusion sampling process toward higher quality infrared generations that better align with input text prompts. Empirically, we find that our approach leads to consistent improvements in generation quality, reducing FID scores on the KAIST Multispectral Pedestrian Detection Benchmark dataset by 10% compared to unguided baseline samples. Our results suggest that inference-time guidance offers a promising direction for bridging the domain gap in low-data infrared settings. \workshoptitle Learning to Sense 1 Introduction Infrared sensors capture thermal information unavailable through conventional RGB cameras, making them essential for applications ranging from autonomous driving to medical imaging. The growing deployment of computer vision systems in such domains has created demand for large-scale synthetic infrared datasets to train robust downstream models such as object detectors and scene classifiers. Diffusion models [ 5 , 20 ] have emerged as a promising approach for generating such synthetic data, offering the potential to create diverse, high-quality images that can augment limited real-world datasets. While the ubiquity of color images allows large-scale diffusion models to be pretrained on billions of RGB images sourced from the web, infrared images remain far less accessible and are typically confined to a limited number of curated public datasets. This scarcity poses significant challenges for training foundation-level infrared diffusion models. Instead, finetuning pretrained RGB models leverages the rich world knowledge encoded from billions of natural images while adapting representations to the infrared modality. To this end, prior works [ 15 , 16 ] have explored techniques to finetune pretrained RGB diffusion models on available public infrared image datasets, albeit with limited physical realism. Such approaches suffer from a lack of contextual grounding; while generated outputs may appear thermodynamically plausible, they often fail to remain consistent with the semantic or contextual cues provided by the conditioning input, whether a textual prompt or RGB image. Other approaches such as PID [ 11 ] train diffusion models directly on limited infrared data with physics-informed constraints, but exhibit inconsistent generation quality and mode collapse, highlighting the pitfalls of direct training in the absence of large-scale pretraining. To address such challenges, we turn to inference-time scaling, an extension of training-time neural scaling laws [ 8 ] commonly observed during the training of deep learning models. Inference-time scaling methods aim to improve the performance of models post hoc by expending additional compute during inference, and have shown promise in their recent application to large language models [ 21 , 23 , 19 ] and diffusion models [ 10 , 14 , 18 ] . Motivated by these advances, we investigate the efficacy of inference-time scaling in improving the quality of synthetic infrared images generated by diffusion models, a direction which, to the best of our knowledge, has not previously been explored. Whereas large-scale pretraining is constrained by infrared data scarcity, we now shift the computational burden from training to generation, thereby improving the visual quality of infrared images produced by diffusion models finetuned on limited data. Building on the inference-time scaling framework of 10 which combines sampling algorithms with verifiers, we extend this approach to the infrared domain and introduce a novel self-supervised verifier designed to evaluate the consistency and realism of synthetic infrared images. In particular, we finetune CLIP [ 13 ] to distinguish true infrared images from their grayscale counterparts, and leverage this model as an inference-time verifier to guide iterative search through the noise space, selecting noise latents that yield the highest-quality generations. Our experiments demonstrate measurable improvements in FID scores, and we further assess how different noise search strategies and verifier training protocols influence inference-time scaling performance. IRScore ( \times 10 ) (ours): 0.5062 IR Similarity: 0.4879 Grayscale Similarity: 0.3866 IRScore ( \times 10 ) (pretrained): 0.1016 (a) Ground Truth IR IRScore ( \times 10 ) (ours): 0.5135 IR Similarity: 0.4213 Grayscale Similarity: 0.3186 IRScore ( \times 10 ) (pretrained): -0.0414 (b) High-Scoring Synthetic IR IRScore ( \times 10 ) (ours): -0.3212 IR Similarity: 0.3189 Grayscale Similarity: 0.3831 IRScore ( \times 10 ) (pretrained): -0.0125 (c) Low-Scoring Synthetic IR Figure 1 : Example verifier scores for a ground truth infrared image and two synthetic images (generated from different random seeds) corresponding to the caption, “ A city street at dusk features tall buildings with illuminated signs, a marked road with directional arrows, and vehicles including a white SUV driving away from the camera. ” IRScore (ours), IR Similarity, and Grayscale Similarity are computed using our finetuned CLIP model, while IRScore (pretrained) relies on a pretrained CLIP model. IRScore , IR Similarity, and Grayscale Similarity correspond to Equation 1 , and its unscaled first and second terms, respectively. 2 Background A naive approach to scaling diffusion models at inference time is to increase the number of denoising steps during image generation. Scaling up denoising steps can improve the visual quality of generated images, albeit with diminishing returns [ 9 ] . Recent works [ 10 , 14 , 18 ] have investigated alternative methods to scaling, motivated by the observation that different noise latents result in varied levels of quality in generated images [ 1 , 12 ] . 10 extends this finding by introducing a scaling approach consisting of two components: verifiers and search algorithms. 2.1 Verifiers Concretely, a verifier is defined as a function \mathcal{V}:\mathbb{R}^{h\times w}\times\mathbb{R}^{d}\to\mathbb{R} which takes in a sample image and an optional condition (e.g., a prompt or image class) and produces a scalar value reflecting the quality of the image, either unconditionally or relative to the provided condition. In the text-to-image setting, 10 explores numerous pretrained models as verifiers, each designed to evaluate different notions of image quality. CLIPScore [ 3 ] is chosen for measuring image-prompt alignment, Aesthetic Score Predictor [ 17 ] for human aesthetic preferences, and ImageReward [ 22 ] for a more comprehensive metric incorporating both aesthetic quality and prompt adherence. Additionally, vision language models (VLMs) such as Gemini-1.5 Flash are employed for their sophisticated text-image understanding and ability to assess generated images across diverse criteria. 2.2 Search Algorithms Formally, a search algorithm is some function f:(\mathcal{V},D_{\theta},\{\mathbf{z}_{1},\dots,\mathbf{z}_{N}\})\mapsto\mathbf{z}^{\ast} which takes as input a verifier \mathcal{V} , a generative diffusion model D_{\theta} , and N candidate noise latents. It then outputs the “best” noise latent \mathbf{z}^{\ast} corresponding to the highest scoring image among N images sampled from D_{\theta} , as determined by \mathcal{V} .