Agentic Flow Steering and Parallel Rollout Search for Spatially Grounded Text-to-Image Generation

Ping Chen, Daoxuan Zhang, Xiangming Wang, Yungeng Liu, Haijin Zeng ∗, Yongyong Chen · 2026

Read Agentic Flow Steering and Parallel Rollout Search for Spatially Grounded Text-to-Image Generation by Ping Chen, Daoxuan Zhang, Xiangming Wang, Yungeng Liu, Haijin Zeng ∗, Yongyong Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18627.txt.

Introduction

Chen and co-authors add a closed feedback loop to text-to-image generation with AFS-Search. Static text encoders reason weakly about spatial relations, and open-loop sampling lets early errors grow. A vision-language model acts as critic, checking draft latents mid-run. Its spatial verdicts steer the flow field directly. Parallel rollouts explore several paths, and reward picks the best one. A Pro variant tops three benchmarks, while a Fast variant keeps most gains at higher speed. The method needs no training and builds on FLUX.1-dev.

Catalog record

CREATORPing Chen, Daoxuan Zhang, Xiangming Wang, Yungeng Liu, Haijin Zeng ∗, Yongyong Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47343
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.18627

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Agentic Flow Steering and Parallel Rollout Search for Spatially Grounded Text-to-Image Generation Ping Chen, Daoxuan Zhang, Xiangming Wang, Yungeng Liu, Haijin Zeng ∗, Yongyong Chen Abstract Precise Text-to-Image (T2I) generation has achieved great success but is hindered by the limited relational reasoning of static text encoders and the error accumulation in open-loop sampling. Without real-time feedback, initial semantic ambiguities during the Ordinary Differential Equation trajectory inevitably escalate into stochastic deviations from spatial constraints. To bridge this gap, we introduce AFS-Search ( A gentic F low S teering and Parallel Rollout Search ), a training-free closed-loop framework built upon FLUX.1-dev. AFS-Search incorporates a training-free closed-loop parallel rollout search and flow steering mechanism, which leverages a Vision-Language Model (VLM) as a semantic critic to diagnose intermediate latents and dynamically steer the velocity field via precise spatial grounding. Complementarily, we formulate T2I generation as a sequential decision-making process, exploring multiple trajectories through lookahead simulations and selecting the optimal path based on VLM-guided rewards. Further, we provide AFS-Search-Pro for higher performance and AFS-Search-Fast for quicker generation. Experimental results show that our AFS-Search-Pro greatly boosts the performance of the original FLUX.1-dev, achieving state-of-the-art results across three different benchmarks. Meanwhile, AFS-Search-Fast also significantly enhances performance while maintaining fast generation speed. Figure 1 : From a visual perspective, our AFS-Search provides a closed-loop generation paradigm to achieve precise spatial grounding generation. 1 Introduction The field of Text-to-Image (T2I) generation has witnessed a paradigm shift with the emergence of Diffusion Models (DMs) [ 14 , 30 , 33 ] and Flow Matching (FM) [ 21 , 20 ] architectures. Models such as SDXL [ 23 ] , FLUX.1 [ 17 ] , and Qwen-Image [ 37 ] have demonstrated an unparalleled ability to synthesize high-fidelity images that capture intricate artistic styles and textures. By leveraging large-scale pre-trained text encoders such as T5 [ 27 ] , CLIP [ 26 ] , these models have moved beyond simple object depiction toward generating complex scenes from natural language descriptions. Recently, VLM-based frameworks such as RPG [ 44 ] , SILMM [ 25 ] , AgentComp [ 46 ] encounter, showing great potential to further complete T2I generation via VLM perception and even agentic actions. Despite these impressive strides, achieving precise spatial grounding and relational reasoning remains a persistent challenge. As shown in Fig. 2 , we identify two primary bottlenecks in conventional T2I pipelines: (1) Static text encoders often exhibit an expressive bottleneck when processing complex relational semantics. Specifically, they struggle to distinguish detailed spatial instructions, which results in underspecified semantic embeddings that fail to capture the nuanced spatial relationships required for accurate image synthesis. (2) Traditional models, as well as recent agent-guided frameworks such as RPG [ 44 ] , Layout-Guidance [ 34 ] , AgentComp [ 46 ] follow an open-loop sampling paradigm. While existing agent-based frameworks enhance spatial control through pre-generation planning of the model, they solve an Ordinary Differential Equation (ODE) along a pre-defined trajectory without any internal feedback. Consequently, even minor semantic ambiguities in the initial phase are irreversibly amplified through the discrete integration steps, ultimately leading to stochastic deviations where the final output fails to satisfy the original spatial constraints. To bridge this gap, we introduce AFS-Search, a training-free closed-loop framework designed to transform T2I generation from a passive sampling process into an active, decision-making procedure. Our core insight is that a generative model should iteratively assess and adjust its generation process rather than producing outputs in a one-shot manner. By integrating a Vision-Language Model (VLM) as a Semantic Critic, our framework enables the system to perceive intermediate generation states and rectify potential errors in real-time. Specifically, we propose Agentic Flow Steering (AFS) a novel steering mechanism that diagnoses semantic drifts at critical timestamps and dynamically steers the velocity field of the flow model via precise spatial grounding leveraging SAM3 [ 1 ] . Additionally, going beyond simple correction, our AFS-Search framework incorporates Parallel Rollout Search (PRS) to effectively navigate the complex latent landscape. At key bifurcation points, the agent performs lookahead simulations by exploring multiple potential trajectories, including corrective steering and random exploration. By evaluating these branches through VLM-guided rewards, the model selects the optimal path that maximizes alignment with the user’s intent. This strategy effectively leverages test-time computation to overcome the inherent randomness of the diffusion process, thereby ensuring robust performance. Further, we provide AFS-Search-Pro for higher performance and AFS-Search-Fast for quicker generation. The main contributions are as follows: Figure 2 : Motivation of our AFS-Search. Open-loop generation follows a fixed, feed-forward sampling trajectory without intermediate feedback or correction while closed-loop generation introduces real-time visual feedback. • We propose AFS-Search, a training-free framework that reformulates T2I generation as as a closed-loop decision-making process instead of a passive sampling task. We provide two versions of AFS-Search for real-time computation and efficiency. • We introduce Agentic Flow Steering mechanism that dynamically rectifies the ODE trajectory via VLM feedback and spatial anchoring via SAM3. • We implement a Parallel Rollout Search strategy using lookahead simulations to effectively resolve spatial and semantic conflicts during sampling. • Experimental results show that our AFS-Search-Pro achieves state-of-the-art results across three benchmarks, while AFS-Search-Fast also significantly enhances performance while meeting the requirements for fast generation. 2 Related Work 2.1 Compositional Text-to-Image Generation T2I generation has achieved remarkable progress, evolving from early GAN-based approaches [ 29 , 47 , 42 ] to the current state-of-the-art Diffusion Models [ 30 , 32 ] and Flow Matching architectures [ 21 , 20 , 17 ] . Despite their ability to synthesize high-quality textures, standard models often struggle with compositional generation . Specifically, correctly binding attributes to objects and strictly following spatial instructions. This limitation stems from the cross-attention mechanism’s tendency to mix semantic information across different spatial regions, leading to attribute leakage or catastrophic neglect of objects. Existing solutions can be broadly categorized into training-based and training-free approaches. Training-based methods [ 48 , 31 ] fine-tune the backbone or introduce additional adapters to enforce spatial constraints. However, in the era of billion-parameter foundation models trained on massive internet-scale datasets, fine-tuning on limited domain-specific data often yields diminishing returns. More critically, which causes the model to lose its open - world generalizability. Consequently, training-free methods have gained prominence. Early works utilized attention manipulation [ 13 , 2 ] to re-weight or mask cross-attention maps. While effective for simple layouts, these heuristic-based methods lack high-level reasoning and often fail in complex scenarios requiring logical planning. Distinct from these, our AFS-Search adopts a test-time search paradigm. We argue that pre-trained models already possess the necessary visual priors; the challenge lies not in learning new features, but in navigating the latent space to locate the correct composition without altering model weights. 2.2 Vision Language Models and Agentic Frameworks The rapid evolution of Large Multimodal Models (LMMs) [ 22 , 6 ] has catalyzed a new wave of Agentic generation frameworks that leverage the reasoning capabilities of LLMs/VLMs to control T2I synthesis. Early works focused on prompt enhancement and layout planning. For instance, DALL-E 3 [ 18 ] and Promptist [ 12 ] employ LLMs to rewrite user queries into descriptive captions. Moving beyond text, LayoutGPT [ 8 ] and VPGen [ 5 ] utilize LLMs to generate

Read the complete text