Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Models
Lit Sin Tan, Junzhe Chen, Xiaolong Fu, Lichen Ma, Junshi Huang, Jianzhong Shi, Yan Li, Lijie Wen · 2026
Read Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Models by Lit Sin Tan, Junzhe Chen, Xiaolong Fu, Lichen Ma, Junshi Huang, Jianzhong Shi, Yan Li, Lijie Wen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15724.txt.
Introduction
Tan and co-authors add test-time learning to unified multimodal image models. Search and sampling tricks improve single outputs but teach the model nothing lasting. Their Meta-TTRL framework tunes model weights at test time instead. Inner monitoring signals from the model itself guide the updates. The method was tested on Janus-Pro-7B, BAGEL, and Qwen-Image. Gains showed on compositional reasoning tasks and several image benchmarks. The team names metacognitive synergy as the key driver. Monitoring signals must match the model's tuning regime.
Catalog record
| CREATOR | Lit Sin Tan, Junzhe Chen, Xiaolong Fu, Lichen Ma, Junshi Huang, Jianzhong Shi, Yan Li, Lijie Wen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32588 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.15724
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Meta-TTRL: A Metacognitive Framework for Self-Improving Test-Time Reinforcement Learning in Unified Multimodal Models Lit Sin Tan, Junzhe Chen, Xiaolong Fu, Lichen Ma, Junshi Huang, Jianzhong Shi, Yan Li, Lijie Wen Abstract Existing test-time scaling (TTS) methods for unified multimodal models (UMMs) in text-to-image (T2I) generation primarily rely on search or sampling strategies that produce only instance-level improvements, limiting the ability to learn from prior inferences and accumulate knowledge across similar prompts. To overcome these limitations, we propose Meta-TTRL , a metacognitive test-time reinforcement learning framework. Meta-TTRL performs test-time parameter optimization guided by model-intrinsic monitoring signals derived from the meta-knowledge of UMMs, achieving self-improvement and capability-level improvement at test time. Extensive experiments demonstrate that Meta-TTRL generalizes well across three representative UMMs, including Janus-Pro-7B, BAGEL, and Qwen-Image, achieving significant gains on compositional reasoning tasks and multiple T2I benchmarks with limited data. We provide the first comprehensive analysis to investigate the potential of test-time reinforcement learning (TTRL) for T2I generation in UMMs. Our analysis further reveals a key insight underlying effective TTRL: metacognitive synergy , where monitoring signals align with the model’s optimization regime to enable self-improvement. † † affiliationnotice: Equal contribution 1 Introduction Unified multimodal models (UMMs) have recently emerged as a powerful paradigm for text-to-image (T2I) generation by integrating multimodal understanding and generation within a single architecture ( 28 ; 5 ; 7 ; 27 ; 24 ; 6 ) . This unified architecture enables coherent semantic grounding and flexible visual synthesis, leading to strong performance in both image quality and compositional fidelity. Figure 1 : Test-Time Reinforcement Learning in UMMs. The rapid progress of UMMs has been largely driven by the success of scaling large pre-trained language models (LLMs) ( 4 ; 1 ) . The strong representational and reasoning capabilities enabled by LLM scaling have inspired the adoption of similar scaling strategies in UMMs ( 5 ; 6 ) . However, this scaling-centric paradigm has a significant downside: the computational cost of training and deploying ever-larger UMMs becomes prohibitively expensive, limiting their practical scalability and accessibility. To mitigate the computational cost of model scaling, recent work has focused on test-time scaling (TTS) methods, which aim to enhance model performance by allocating additional computational resources during test time. Recent studies have shown that TTS can improve T2I generation performance in UMMs ( 21 ; 11 ; 15 ; 23 ) . TTS approaches for T2I generation in UMMs can be generally categorized into two paradigms. (1) Parallel sampling-based scaling ( 21 ; 11 ) , where multiple candidate images are generated through sampling and the best one is selected via Best-of- N selection based on verification scores. Candidate ranking in this paradigm generally relies on two types of verification scores: external reward models ( 11 ) , which employ outcome reward models or process reward models to assess the alignment between the generated image and the prompt; and self-verification ( 21 ) , in which the UMM is specifically trained to serve as the verifier, thereby acquiring chain-of-thought-based verification capability. (2) Sequential iterative scaling ( 15 ; 23 ) , where the UMM progressively refines outputs across multiple inference rounds through reasoning, evaluation, and refinement steps, enabling iterative improvement based on intermediate results. However, current TTS approaches for T2I generation in UMMs improve generation performance primarily through test-time sampling or search strategies while keeping the model parameters fixed. As a result, the additional computation yields only instance-level improvements , and the UMM fails to accumulate knowledge from prior inference experiences. In other words, any performance gain is ephemeral : when encountering similar prompts in subsequent queries, the UMM often exhibits the same failure modes and must repeat the entire TTS procedure again. This observation naturally raises a key question: Can the experience obtained during test-time scaling be leveraged for learning? A promising direction is test-time reinforcement learning (TTRL), which leverages test-time experience as learning signals to update model parameters at test time. TTRL has recently shown strong potential in language reasoning tasks ( 30 ) and vision–language understanding tasks ( 19 ) . However, despite its success in these tasks, TTRL for T2I generation in UMMs remains largely unexplored. This leads to a key question: Can reinforcement learning enable effective and robust test-time learning for T2I generation in UMMs without relying on external reward models? To answer this question, we propose Meta-TTRL , a metacognitive test-time reinforcement learning framework for T2I generation in UMMs. As illustrated in Figure 1 , Meta-TTRL extends TTS by leveraging the experience obtained during TTS as learning signals to update model parameters via reinforcement learning. The framework introduces a two-level metacognitive architecture in which the UMM operates at two levels: an object-level generator produces candidate images, while a meta-level introspector constructs structured evaluation rubrics and monitors generation outputs. The model-intrinsic monitoring signals, derived from the meta-knowledge of UMMs, eliminate the need for external reward models while providing robust supervision and enabling self-improvement. Through the interaction between meta-level monitoring and object-level policy optimization, Meta-TTRL forms a monitoring–control loop that enables capability-level improvement, allowing performance gains to persist when the model encounters similar prompts. Beyond proposing a new framework, we aim to better understand the potential of TTRL for T2I generation in UMMs. Through comprehensive analysis, we identify a key insight underlying effective TTRL, referred to as metacognitive synergy . Specifically, robust and effective TTRL emerges when monitoring signals are well aligned with the model’s optimization regime and model’s capacity for self-improvement. This finding suggests that the effectiveness of TTRL is determined less by the absolute capability or scale of introspectors, and more by how well monitoring signals align with the model’s optimization regime. Our contributions can be summarized as: • Meta-TTRL Framework. We propose Meta-TTRL , the first test-time reinforcement learning framework for T2I generation in UMMs. Meta-TTRL leverages model-intrinsic monitoring signals derived from the meta-knowledge of UMMs to guide test-time policy optimization, enabling self-improvement. • Comprehensive Analysis of TTRL. We conduct analyses to study the potential of TTRL from three aspects: external introspector (E-TTRL) , RL leakage , and alternative monitoring signals . These analyses reveal a key insight underlying effective TTRL, termed metacognitive synergy : robust and effective TTRL emerges when monitoring signals are well aligned with the model’s optimization regime. • Strong Empirical Improvements. Extensive experiments show that Meta-TTRL consistently improves T2I generation performance across three representative UMMs: Janus-Pro-7B, BAGEL, and Qwen-Image. Meta-TTRL yields significant improvements in compositional reasoning tasks (e.g., shape, texture, and spatial reasoning), with relative gains exceeding 50% for weaker base models while also providing consistent benefits for stronger models. In addition, the improvements generalize well to out-of-distribution settings, including cross-benchmark evaluations, indicating that Meta-TTRL does not overfit to specific benchmarks. 2 Related Work Unified Multimodal Models (UMMs). Unified multimodal models (UMMs) jointly support multimodal understanding and generation. Autoregressive methods extend the next-token prediction paradigm to jointly model text and discrete visual tokens ( 20 ; 24 ; 5 ) . Diffusion methods couple language models with diffusion modules for image generation ( 8 ; 27 ) . Unified transformer architectures integrate language modeling and diffusion within single architectures ( 29 ; 7 ) . Our framework generalizes across all three architectural paradigms and is evaluated on three representative UMMs,