CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration

Seyed Amir Kasaei [email protected], Ali Aghayari [email protected], Arash Marioriyad [email protected], Niki Sepasian [email protected], Shayan Baghayi Nejad [email protected], MohammadAmin Fazli [email protected], Mahdieh Soleymani Baghshah [email protected], Mohammad Ho · 2025

Read CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration by Seyed Amir Kasaei [email protected], Ali Aghayari [email protected], Arash Marioriyad [email protected], Niki Sepasian [email protected], Shayan Baghayi Nejad [email protected], MohammadAmin Fazli [email protected], Mahdieh Soleymani Baghshah [email protected], Mohammad Ho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17458.txt.

Introduction

Seyed Amir Kasaei and seven co-authors help text-to-image models honor complex prompts. Their 2025 arXiv paper notes that diffusion models draw fine images but often botch object relations, attributes, and layout. A growing fix tunes the starting noise at inference time under reward scores, with no fine-tuning of the model itself.

Used alone, each fix has a flaw: pure optimization can stall from a poor start, while pure exploration may need a huge sample count. Single reward scores also miss parts of compositionality, and ad hoc mixes guide weakly. Their framework, CARINOX, joins noise optimization with exploration and picks rewards by category, grounded in how well scores track human judgment. Tests on two benchmarks show steadier alignment for tangled prompts. The method leaves the base model untouched, which keeps it portable.

Catalog record

CREATORSeyed Amir Kasaei [email protected], Ali Aghayari [email protected], Arash Marioriyad [email protected], Niki Sepasian [email protected], Shayan Baghayi Nejad [email protected], MohammadAmin Fazli [email protected], Mahdieh Soleymani Baghshah [email protected], Mohammad Ho
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-17458

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Exploration Seyed Amir Kasaei [email protected], Ali Aghayari [email protected], Arash Marioriyad [email protected], Niki Sepasian [email protected], Shayan Baghayi Nejad [email protected], MohammadAmin Fazli [email protected], Mahdieh Soleymani Baghshah [email protected], Mohammad Hossein Rohban [email protected] Abstract Text-to-image diffusion models, such as Stable Diffusion, can produce high-quality and diverse images but often fail to achieve compositional alignment , particularly when prompts describe complex object relationships, attributes, or spatial arrangements. Recent inference-time approaches address this by optimizing or exploring the initial noise under the guidance of reward functions that score text–image alignment—without requiring model fine-tuning. While promising, each strategy has intrinsic limitations when used alone: optimization can stall due to poor initialization or unfavorable search trajectories, whereas exploration may require a prohibitively large number of samples to locate a satisfactory output. Our analysis further shows that neither single reward metrics nor ad-hoc combinations reliably capture all aspects of compositionality, leading to weak or inconsistent guidance. To overcome these challenges, we present C ategory- A ware R eward-based I nitial N oise O ptimization and E X ploration ( CARINOX ), a unified framework that combines noise optimization and exploration with a principled reward selection procedure grounded in correlation with human judgments. Evaluations on two complementary benchmarks—covering diverse compositional challenges—show that CARINOX raises average alignment scores by +16% on T2I-CompBench++ and +11% on the HRS benchmark, consistently outperforming state-of-the-art optimization and exploration-based methods across all major categories, while preserving image quality and diversity. The project page is available at this URL . Figure 1 : Qualitative results on T2I-CompBench++ , showing that CARINOX faithfully captures compositional details such as counts, spatial arrangements, and attribute bindings. 1 Introduction Text-to-image (T2I) diffusion models, such as Stable Diffusion (SD) ( Rombach et al. 2022 ; Podell et al. 2023 ) and DALL-E ( Ramesh et al. 2022 ) , have garnered substantial attention for their ability to synthesize high-quality images from natural language prompts through iterative denoising and cross-modal attention mechanisms. These models have been adopted in a wide range of applications, including image editing ( Huang et al. 2024b ; Kawar et al. 2023 ; Liu et al. 2024a ; Mou et al. 2024 ) , data augmentation ( Li et al. 2024c ; Xiao et al. 2023 ; Feng et al. 2023a ) , medical imaging ( Huang et al. 2024a ; Li et al. 2024b ; Khader et al. 2023 ; Lin et al. 2024a ) , and marketing ( Shilova et al. 2023 ; Yang et al. 2024 ) . Despite their versatility and impressive generation capabilities, T2I diffusion models often exhibit notable failures in compositional alignment ( Huang et al. 2025 ; Bakr et al. 2023 ; Ghosh et al. 2024 ) . These failures manifest in various forms, including entity omission ( Chefer et al. 2023b ; Sueyoshi & Matsubara 2024a ; Zhang et al. 2024a ; Liu et al. 2022 ; Kim et al. 2023 ) , incorrect attribute binding ( Feng et al. 2023b ; Li et al. 2023c ; Rassin et al. 2024 ; Wang et al. 2024 ) , misrepresentation of spatial relationships ( Chatterjee et al. 2024 ; Gokhale et al. 2022 ; Chen et al. 2024 ) , and numeracy errors ( Binyamin et al. 2024 ; Zafar et al. 2024 ; Kang et al. 2023b ) . To address compositional generation failures, several studies have explored fine-tuning-based approaches. While effective, such methods are often computationally expensive and time-consuming. In response, a range of inference-time techniques has emerged, aiming to improve generation quality without modifying the underlying model. A similar trend has been observed in large language models (LLMs), where recent work enhances reasoning capabilities by employing verifiers—such as reward functions—during inference rather than through fine-tuning. Within the T2I domain, a subset of inference-time methods focuses on leveraging the initial noise to improve alignment. These approaches fall into two main categories: optimization-based methods, such as ReNO ( Eyring et al. 2024 ) and InitNo ( Guo et al. 2024b ) , which iteratively refine the initial noise to maximize alignment based on a reward signal; and exploration-based methods, including ImageSelect ( Karthik et al. 2023 ) , SeedSelect ( Samuel et al. 2024b ) , SemI ( Mao et al. 2024 ) , ParticleFiltering ( Liu et al. 2024b ) , and ReliableRandomSeeds ( Li et al. 2024a ) , which evaluate multiple noise samples and select the one yielding the best result. In both settings, reward functions guide the process by scoring how well each candidate image matches the input prompt. Despite recent progress, existing approaches face two critical challenges that we address in this work. First, both continuous noise optimization and discrete noise selection strategies suffer from inherent limitations when used in isolation. Optimization methods are sensitive to the choice of initial noise and may fail to align the generated image with the prompt due to poor initialization or unfavorable optimization trajectories—even when the starting image appears qualitatively plausible (see Figure 2(a) ). In contrast, exploration-based methods are limited by the nature of their search process: they typically sample from a fixed set of candidates and evaluate each independently, often requiring many trials to find a well-aligned output, particularly in the high-dimensional latent space of diffusion models (see Figure 2(b) ). These limitations are analyzed in more detail in Section 3 . Second, the choice of reward function is crucial for guiding generation, yet remains underexplored. Many existing works adopt commonly used metrics without accounting for the specific challenges of compositionality, such as spatial reasoning, entity binding, or numeracy. As a result, the reward signal may be weak or misaligned, reducing the effectiveness of both optimization and exploration. To overcome these limitations, we propose CARINOX, a novel framework that integrates both noise optimization and exploration strategies with a carefully selected reward function to improve compositional alignment in T2I generation. CARINOX addresses the shortcomings of existing methods by combining continuous optimization of initial noise with a targeted discrete exploration strategy, effectively reducing the risk of poor optimization paths and the inefficiency of blind sampling. To support this process, we systematically derive a robust combination of reward metrics through an empirical correlation study against human judgments, ensuring that the guidance used during generation is aligned with compositional quality. Through this design, CARINOX unifies the strengths of both optimization and exploration while grounding the reward function in a principled, data-driven selection process tailored to compositional challenges. We evaluate CARINOX on two widely used benchmarks—T2I-CompBench++ Huang et al. 2025 and HRS ( Bakr et al. 2023 ) —covering a broad spectrum of compositional challenges. Across both datasets, CARINOX consistently improves over the underlying backbones. On T2I-CompBench++ , it raises the average performance of SD-Turbo from 0.39 to 0.57 , SDXL-Turbo from 0.41 to 0.57 , and PixArt- \alpha from 0.35 to 0.58 , with the strongest gains in texture, numeracy, and spatial reasoning. On the HRS benchmark , it further enhances all three backbones, delivering mean improvements of +0.18 on SD-Turbo, +0.16 on SDXL-Turbo, and +0.23 on PixArt- \alpha , and setting new highs in creativity, style, and visual writing. Notably, these gains are achieved while preserving image quality and diversity, showing

Read the complete text