Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners
Qingyang Liu, Bingjie Gao, Canmiao Fu, Zhipeng Huang, Chen Li, Feng Wang, Shuochen Chang, Shaobo Wang, Yali Wang, Keming Ye, Jiangtong Li, Li Niu · 2026
Read Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners by Qingyang Liu, Bingjie Gao, Canmiao Fu, Zhipeng Huang, Chen Li, Feng Wang, Shuochen Chang, Shaobo Wang, Yali Wang, Keming Ye, Jiangtong Li, Li Niu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14709.txt.
Introduction
Liu and team tackle two bottlenecks in unified multimodal models that turn any input into images. One is attention entanglement in blind planning for complex prompts. The other is weak visual refinement from unstructured feedback. Their framework lets a model switch on its own between direct generation for simple cases, self-review for quality fixes, and multi-step planning for complex scenes. A data pipeline with over fifty thousand samples and a two-stage training strategy support the switch. Gains over baselines are claimed.
Catalog record
| CREATOR | Qingyang Liu, Bingjie Gao, Canmiao Fu, Zhipeng Huang, Chen Li, Feng Wang, Shuochen Chang, Shaobo Wang, Yali Wang, Keming Ye, Jiangtong Li, Li Niu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44490 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14709
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners Qingyang Liu, Bingjie Gao, Canmiao Fu, Zhipeng Huang, Chen Li, Feng Wang, Shuochen Chang, Shaobo Wang, Yali Wang, Keming Ye, Jiangtong Li, Li Niu Abstract Recent unified models integrate multimodal understanding and generation within a single framework. However, an “understanding-generation gap” persists, where models can capture user intent but often fail to translate this semantic knowledge into precise pixel-level manipulation. This gap results in two bottlenecks in anything-to-image task (X2I): the attention entanglement bottleneck , where blind planning struggles with complex prompts, and the visual refinement bottleneck , where unstructured feedback fails to correct imperfections efficiently. In this paper, we propose a novel framework that empowers unified models to autonomously switch between generation strategies based on instruction complexity and model capability. To achieve this, we construct a hierarchical data pipeline that constructs execution paths across three adaptive modes: direct generation for simple cases, self-reflection for quality refinement, and multi-step planning for decomposing complex scenarios. Building on this pipeline, we contribute a high-quality dataset with over 50,000 samples and implement a two-stage training strategy comprising SFT and RL. Specifically, we design step-wise reasoning rewards to ensure logical consistency and intra-group complexity penalty to prevent redundant computational overhead. Extensive experiments demonstrate that our method outperforms existing baselines on X2I, achieving superior generation fidelity among simple-to-complex instructions. The code is released at GitHub . 1 Introduction Figure 1 : Comparison of conventional paradigms and our Self-Adaptive Interleaved Visual Reasoners. Left: Existing methods struggle with “blind” planning and unstructured reflection due to understanding-generation gap. Right: Our approach adopts adaptive strategies: Mode 1 decomposes complex prompts via interleaved generation; Mode 2 employs structured reflection for interleaved visual refinement. Recent advancements in unified large multimodal models (unified models) ( 64 ; 62 ) have integrated multimodal understanding and generation within a single framework. Inspired by the success of Chain-of-Thought (CoT) reasoning, recent studies ( 76 ; 81 ; 22 ) have incorporated multimodal CoT reasoning into these unified architectures. This integration enables complex anything-to-image (X2I) tasks ( 42 ; 47 ; 41 ; 82 ; 80 ) , requiring models to generate or edit images conditioned on diverse multimodal inputs ( e.g. , text, layouts, sketches). Although these models demonstrate promising visual reasoning and generative capabilities, a critical disparity remains between their understanding potential and actual generation fidelity ( 5 ) . Specifically, while current unified models exhibit strong visual comprehension, their ability to execute precise, instruction-following image editing often lags behind. We term this issue the “ understanding-generation gap ”, which becomes evident in complex X2I scenarios: the model “ knows ” the user’s intent but fails to “ faithfully translate ” this semantic knowledge into accurate pixel-level manipulation. Driven by this understanding-generation gap, unified models face two primary challenges when executing complex tasks: the attention entanglement bottleneck ( 49 ; 28 ) and the visual refinement bottleneck ( 65 ) . The attention entanglement bottleneck stems from the complexity of user prompts, which act as a barrier to direct, single-pass synthesis and require decomposing instructions into step-by-step edits. Meanwhile, the visual refinement bottleneck arises from unavoidable imperfections in single-step pixel synthesis, creating a need for progressive visual refinement. To mitigate the attention entanglement bottleneck, recent approaches formulate detailed textual plans before execution ( 53 ; 75 ) . However, these methods lack verification mechanisms; the reasoning process remains “blind” to the model’s actual generative limits, often yielding unexecutable plans that fail to handle complex, multi-step editing. To handle the visual refinement bottleneck, several methods employ critiques of initial outputs to guide subsequent refinement ( 18 ; 85 ) . These methods typically rely on switching between multiple distinct models, which incurs high computational costs and obstructs automated, progressive self-improvement. Recent efforts attempt to iteratively generate and critique images using a unified model within a feedback loop ( 17 ) . However, such unstructured critiques mix error analysis and improvement suggestion together, making it difficult to resolve compound errors efficiently. Crucially, no existing work has successfully targeted both bottlenecks simultaneously within a unified framework. To overcome these two bottlenecks, we propose a Self-adaptive Interleaved Reasoner, which equips the unified models with native interleaved generation and dynamic self-evaluation capabilities, allowing it to autonomously switch strategies based on instruction complexity and model capability. Fig. 1 illustrates two interleaved reasoning modes of our reasoner. Specifically, for complex multi-intent instructions, the reasoner utilizes textual planning and interleaved generation to decompose tasks step-by-step. When initial outputs fail to meet quality standards, the reasoner switches to a self-reflection mode for progressive visual refinement. To develop this adaptive capability, we propose a data pipeline driven by a hierarchical escalation mechanism. Given a raw X2I ( 52 ) input, we first employ the baseline unified models to directly generate the image. We then use an Analyzer , Qwen3vl-235B ( 1 ) , as an independent critic to evaluate the output’s alignment with the user instruction and reference inputs. If the initial generation is unsatisfactory, the Analyzer diagnoses specific errors in the faulty image and synthesizes a reflection prompt. Guided by this reflection, a Generator , Gemini-3-Pro-Image ( 16 ) , attempts to rectify the output. If the task remains unresolved after the maximum iteration limit, the Analyzer identifies reason for such failure. Cases identified as having excessive prompt complexity escalate to the multi-step planning phase, while others ( e.g. , those requiring specialized knowledge) are filtered out. For multi-step planning phase, the LMM decomposes the complex instruction into sequential sub-tasks, enabling step-by-step generation and intermediate evaluation until the objective is met. Following strict human verification, we obtain a high-fidelity interleaved dataset with over 50,000 samples that integrates generation, reflection, planning, and evaluation across three adaptive modes. Leveraging this constructed dataset, we implement a two-stage training paradigm. First, we apply Supervised Fine-Tuning (SFT) to adapt the model to the interleaved reasoning syntax and basic instruction following. Subsequently, we employ Reinforcement Learning (RL) with Group Relative Policy Optimization algorithm (GRPO) to optimize the model’s strategic planning, ensuring it selects the most effective generation mode for varying instruction complexities. Beyond standard format and outcome rewards assessed by an LMM, we introduce a step-wise reasoning reward and an intra-group complexity penalty. The step-wise reasoning reward verifies the logical consistency of intermediate steps, utilizing the LMM to validate each textual output. To prevent inefficient over-reasoning, the intra-group complexity penalty favors simpler generation modes when they yield results equivalent to complex ones, thereby balancing generation quality with computational efficiency. Extensive experiments and visualization results confirm that our method outperforms existing baselines (EUM3.5 ( 7 ) , etc. ) among text-to-image, image editing, and X2I tasks. Our contributions can be summarized as: • We propose a unified framework with interleaved plan or reflect, resolving the understanding-generation gap of unified models by addressing attention entanglement and visual refinement bottlenecks. • We design a hierarchical data synthesis pipeline that constructs execution paths across direct, reflective, and multi-step planning modes. Based on this pipeline, we contribute a large-scale high-fidelity dataset to enable adaptive X2I generation. • We implement a two-stage training strategy with SFT and RL. With our proposed step-wise reasoning reward and intra-group complexity penalty, we ensure logical correctness while