TextDiffuser-RL: Efficient and Robust Text Layout Optimization for High-Fidelity Text-to-Image Synthesis

Kazi Mahathir Rahman 1 , Showrin Rahman 2 , Sharmin Sultana Srishty 3 1 BRAC University, Dhaka, Bangladesh [email protected] , [email protected] , [email protected] · 2025

Read TextDiffuser-RL: Efficient and Robust Text Layout Optimization for High-Fidelity Text-to-Image Synthesis by Kazi Mahathir Rahman 1 , Showrin Rahman 2 , Sharmin Sultana Srishty 3 1 BRAC University, Dhaka, Bangladesh [email protected] , [email protected] , [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19291.txt.

Introduction

Kazi Mahathir Rahman, Showrin Rahman, and Sharmin Sultana Srishty speed up text-in-image generation by letting reinforcement learning lay out the words, in a paper from 2025. Posters and ads need crisp, well-placed text inside AI images. The leading tool, TextDiffuser-2, plans text boxes with a heavy model that cannot even run on a plain CPU.

Their two-stage pipeline swaps that planner for a small learning agent. It predicts bounding boxes fast and with fewer overlaps, then a diffusion model paints the image. Quality matched the big system on placement and synthesis tests. Runtime, though, was about forty-two times faster, and inference needed only two megabytes of memory. The authors pitch the method for design tools on modest hardware.

Catalog record

CREATORKazi Mahathir Rahman 1 , Showrin Rahman 2 , Sharmin Sultana Srishty 3 1 BRAC University, Dhaka, Bangladesh [email protected] , [email protected] , [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19291

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TextDiffuser-RL: Efficient and Robust Text Layout Optimization for High-Fidelity Text-to-Image Synthesis Kazi Mahathir Rahman 1 , Showrin Rahman 2 , Sharmin Sultana Srishty 3 1 BRAC University, Dhaka, Bangladesh [email protected] , [email protected] , [email protected] Abstract Text-embedded image generation plays a critical role in industries such as graphic design, advertising, and digital content creation. Text-to-Image generation methods leveraging diffusion models, such as TextDiffuser-2 [ 26 ] , have demonstrated promising results in producing images with embedded text. TextDiffuser-2 effectively generates bounding box layouts that guide the rendering of visual text, achieving high fidelity and coherence. However, existing approaches often rely on resource-intensive processes and are limited in their ability to run efficiently on both CPU and GPU platforms. To address these challenges, we propose a novel two-stage pipeline that integrates reinforcement learning (RL) for rapid and optimized text layout generation with a diffusion-based image synthesis model. Our RL-based approach significantly accelerates the bounding box prediction step while reducing overlaps, allowing the system to run efficiently on both CPUs and GPUs. Extensive evaluations demonstrate that our framework maintains or surpasses TextDiffuser-2’s quality in text placement and image synthesis, with markedly faster runtime and increased flexibility. Extensive evaluations demonstrate that our framework maintains or surpasses TextDiffuser-2’s quality in text placement and image synthesis, with markedly faster runtime and increased flexibility. Our approach has been evaluated on the MARIOEval [ 27 ] benchmark, achieving OCR and CLIPScore metrics close to state-of-the-art models, while being 97.64% more faster and requiring only 2MB of memory to run. Keywords: Reinforcement learning, DDPG, PPO, SAC, Textdiffuser, text2image, Text refinement 1 Introduction Text-embedded image generation is an emerging research problem that lies at the intersection of computer vision and natural language processing. It focuses on generating images that contain textual content rendered in a visually coherent and semantically accurate manner. This task is crucial for several real-world applications, including poster design, educational content, advertisements, digital storytelling, and visual communication. The ability to control both the textual content and its layout within the image is essential, as even minor spelling or placement errors can compromise the utility and credibility of the generated output. Recent advancements in diffusion-based models have significantly improved the quality of text-to-image generation. Notable among them is TextDiffuser-2 [ 26 ] , which introduces a two-stage diffusion pipeline to synthesize text-embedded images. While effective in generating visually compelling outputs, these models suffer from layout collisions, overlapping bounding boxes, and spelling inaccuracies. Additionally, the computational overhead associated with generating these high-resolution images poses challenges for real-time or resource-constrained deployments. Alternative approaches such as SA-OcrPaint [ 28 ] attempt to mitigate spelling errors and layout inaccuracies through iterative OCR-guided layout refinement and inpainting. However, the reliance on iterative post-processing makes these models inefficient and unsuitable for time-sensitive applications. Moreover, these approaches lack flexibility in dynamically adjusting the layout based on textual or visual constraints, which limits their generalizability. To address these challenges, we explore the potential of reinforcement learning (RL) as a mechanism for layout optimization in text-embedded image generation. RL provides a natural framework for learning policies that maximize long-term rewards based on intermediate feedback signals. In our context, these signals are derived from OCR-based spelling accuracy and CLIP-based perceptual similarity. Unlike heuristic or optimization-based methods, RL can learn adaptable policies that generalize across diverse text prompts and image layouts. Our key insight is that integrating an RL-based layout optimization stage into the diffusion pipeline can significantly enhance text fidelity and visual coherence while reducing computational cost. By decoupling layout planning from image generation, we can leverage a lightweight RL agent to generate optimal bounding box placements before invoking the diffusion model. This two-stage strategy enables faster and more accurate synthesis without sacrificing output quality. Motivated by these observations, we introduce TextDiffuser-RL , a novel framework that integrates reinforcement learning for real-time layout optimization with a diffusion-based image synthesis backbone. As illustrated in Figure 1 , Our framework leverages a two-stage process for optimization, utilizing a diffusion-based image synthesis backbone. In the initial stage, a reinforcement learning agent called GlyphEnv is employed. GlyphEnv is trained within a custom environment to generate non-overlapping and spatially aware bounding boxes for each text prompt. Additionally, GlyphEnv simulates the layout planning process and assesses the quality of each layout through a reward function that considers OCR accuracy and spatial alignment metrics. Figure 1 : Overview of the Model Architecture, the diffusion-based synthesis process with GlyphEnv optimizing non-overlapping bounding box generation, layout planning and quality assessment based on OCR accuracy and spatial alignment. In the second stage, the generated layout is passed to a modified Stable Diffusion model equipped with a VEA decoder, which functions similarly to TextDiffuser-2. This architecture ensures that the diffusion model receives well-structured input, thereby improving text rendering, reducing spelling mistakes, and minimizing layout collisions. Our method maintains visual fidelity while achieving substantial gains in efficiency. We evaluate TextDiffuser-RL on the MARIO-Eval benchmark and compare its performance with existing state-of-the-art methods. Experimental results demonstrate that our method is 97.64% faster and uses only 2 MB of memory, while maintaining competitive OCR accuracy and CLIPScore. These results confirm the effectiveness of combining layout-aware reinforcement learning with generative diffusion models for text-embedded image synthesis. In this paper, we propose a novel framework, TextDiffuser-RL , which advances the field of text-embedded image generation by introducing the following key contributions: • We propose TextDiffuser-RL , the first framework to integrate reinforcement learning-based layout optimization with diffusion-based text-embedded image generation. • We introduce GlyphEnv , a novel RL environment tailored for generating non-overlapping and visually coherent text layouts using PPO, DDPG, and SAC algorithms. • We show that TextDiffuser-RL significantly reduces runtime and memory usage while maintaining text accuracy, as validated on the MARIO-Eval benchmark. 2 Related Work Text-embedded image generation remains a challenging task, particularly when aiming for accurate and visually coherent rendering of textual content within images. The MARIO-Eval [ 27 ] benchmark has become a critical standard for evaluating progress in this domain. Several recent works have made significant strides by innovating on both text layout optimization and image synthesis techniques. UDiffText [ 1 ] is studied, a framework consisting of a diffusion model for generating text images based on a pre-trained model. UDiffText, a framework that utilizes a pre-trained diffusion model for text image generation. By designing a lightweight character-level text encoder and fine-tuning the diffusion model with large-scale datasets, UDiffText achieves high sequence accuracy in synthesizing text within arbitrary images. The approach demonstrates superior performance in text-centric image synthesis and scene text editing. Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering a customized text encoder that fine-tunes the character-aware ByT5 model using paired glyph-text data. This encoder aligns closely with visual text signals, significantly enhancing text rendering accuracy. When integrated with the SDXL model, Glyph-ByT5 improved text rendering accuracy from less than 20 percent to nearly 90 percent on design image benchmarks, showcasing its effectiveness in handling complex text rendering tasks. [ 2 ] TextDiffuser [ 26 ] was among the pioneering approaches to integrate a two-stage diffusion model specialized for text-rich images. Its pipeline employs a transformer-based layout planner to generate keyword bounding boxes, followed by a diffusion U-Net conditioned on this layout. TextDiffuser introduced the

Read the complete text