SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning
Jichao Wang Liuyang Bian Yufeng Zhou Han Xiao, Yue Pan Guozhi Wang Hao Wang Zhaoxiong Wang, Yafei Wen Xiaoxin Chen Shuai Ren Lingfang Zeng · 2026
Read SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning by Jichao Wang Liuyang Bian Yufeng Zhou Han Xiao, Yue Pan Guozhi Wang Hao Wang Zhaoxiong Wang, Yafei Wen Xiaoxin Chen Shuai Ren Lingfang Zeng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22558.txt.
Introduction
Wang and 11 colleagues present SOLAR-RL, a training scheme for multimodal language-model agents navigating graphical interfaces. Offline reinforcement learning leans on static step-level data and misses whole-trajectory meaning such as task completion and execution quality. Online learning captures long-term dynamics at high interaction cost and environmental risk. Their semi-online approach folds global trajectory insight into offline learning instead. It rebuilds rollout candidates from static data, spots the first failure point with per-step validity signals, and assigns dense step rewards shaped to trajectory-level quality.
Catalog record
| CREATOR | Jichao Wang Liuyang Bian Yufeng Zhou Han Xiao, Yue Pan Guozhi Wang Hao Wang Zhaoxiong Wang, Yafei Wen Xiaoxin Chen Shuai Ren Lingfang Zeng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42908 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22558
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SOLAR-RL: Semi-Online Long-horizon Assignment Reinforcement Learning Jichao Wang Liuyang Bian Yufeng Zhou Han Xiao, Yue Pan Guozhi Wang Hao Wang Zhaoxiong Wang, Yafei Wen Xiaoxin Chen Shuai Ren Lingfang Zeng Abstract As Multimodal Large Language Models (MLLMs) mature, GUI agents are evolving from static interactions to complex navigation. While Reinforcement Learning (RL) has emerged as a promising paradigm for training MLLM agents on dynamic GUI tasks, its effective application faces a dilemma. Standard Offline RL often relies on static step-level data, neglecting global trajectory semantics such as task completion and execution quality. Conversely, Online RL captures the long-term dynamics but suffers from high interaction costs and potential environmental instability. To bridge this gap, we propose SOLAR-RL ( S emi- O nline L ong-horizon A ssignment R L). Instead of relying solely on expensive online interactions, our framework integrates global trajectory insights directly into the offline learning process. Specifically, we reconstruct diverse rollout candidates from static data, detect the first failure point using per-step validity signals, and retroactively assign dense step-level rewards with target-aligned shaping to reflect trajectory-level execution quality—effectively simulating online feedback without interaction costs. Extensive experiments demonstrate that SOLAR-RL significantly improves long-horizon task completion rates and robustness compared to strong baselines, offering a sample-efficient solution for autonomous GUI navigation. 1 Introduction The development of autonomous agents capable of mastering Graphical User Interfaces (GUIs) is a pivotal frontier in multimodal AI. Unlike traditional agents confined to specific APIs or DOM-tree parsing, modern GUI agents powered by Multimodal Large Language Models (MLLMs) operate directly on pixel-level visual inputs, promising universal applicability across diverse operating systems Qin et al. 2025 . While agents like CogAgent Hong et al. 2024 have demonstrated proficiency in perceiving complex interfaces and executing atomic actions, a significant gap remains between "perceiving" an interface and "completing" long-horizon, multi-step workflows in real-world environments. Figure 1: Comparison of RL paradigms for GUI agents. (Top Left) Standard Offline RL is limited by fragmented step-level data, leading to temporal myopia and loss of global context. (Top Right) Online RL captures dynamics but suffers from instability and prohibitive interaction costs. (Bottom) Our SOLAR-RL bridges this gap by retrofitting global trajectory insights into offline data. It utilizes trajectory reconstruction and retroactive credit assignment via failure-point detection, combined with target-aligned reward shaping, to simulate pseudo-online feedback, ensuring stable long-horizon optimization. Current state-of-the-art (SOTA) approaches primarily rely on Supervised Fine-Tuning (SFT) or Behavior Cloning (BC) from expert demonstrations Rawles et al. 2023 ; Li et al. 2024 . While SFT effectively initializes agents with basic semantic understanding, it suffers heavily from covariate shift Ross et al. 2011 . When an agent encounters states deviating slightly from the training distribution—a frequent occurrence in dynamic GUIs—it lacks recovery mechanisms, leading to compounding errors. Consequently, recent works have pivoted towards Reinforcement Learning (RL) to endow agents with planning and exploration capabilities Bai et al. 2024 ; Wang et al. 2025 . However, as illustrated in Figure 1 , applying RL to GUI automation presents a structural dilemma, particularly for long-horizon tasks. Online RL, while theoretically capable of capturing dynamic environmental feedback, suffers from severe instability and high variance when scaling to long trajectories. The prohibitive interaction costs and sparse reward signals in 30+ step workflows often lead to optimization failure before a successful policy can be learned Bai et al. 2024 . Conversely, Offline RL circumvents interaction risks but is plagued by temporal myopia . By constraining learning to fragmented, step-level transitions in a static dataset, standard offline methods lose the global context required for long-term planning, making them susceptible to compounding errors Levine et al. 2020 . Critically, both paradigms grapple with the Credit Assignment Problem (CAP) Lu et al. 2025b . In long-horizon GUI navigation, a sparse binary "success/failure" signal at the trajectory’s end is insufficient to assign credit to intermediate reasoning steps. Consequently, gradients vanish or become noisy, leaving agents unable to distinguish critical decisions from irrelevant actions. This challenge motivates the need for a paradigm that retains the stability of offline learning while incorporating trajectory-level signals typically available only in online interaction. The Semi-Online RL paradigm illustrated in Figure 1 (Right) offers a compromise. Our proposed SOLAR-RL is a concrete instantiation of this paradigm. SOLAR-RL ( S emi- O nline L ong-horizon A ssignment R L) synthesizes pseudo-online feedback from offline data to address the long-horizon credit-assignment problem. Specifically, we reconstruct diverse rollout candidates from static data and evaluate per-step validity to detect the first failure point where execution breaks down. We then perform retroactive credit assignment by granting positive rewards only to the valid prefix before the failure point while penalizing invalid steps after breakdown. Finally, we apply target-aligned reward shaping to align the total shaped return with trajectory-level execution quality, producing dense and stable training signals without any environment interaction. Our main contributions are summarized as follows: • We propose SOLAR-RL, a framework that bridges the gap between offline training stability and online exploration by simulating dynamic feedback mechanisms within static datasets. • We introduce a trajectory-aware reward shaping mechanism that addresses the credit assignment problem by performing retroactive credit assignment via failure-point detection and target-aligned reward shaping, distilling trajectory-level execution quality into dense step-level rewards. • We achieve competitive performance with online/SFT baselines without interaction, specifically demonstrating superior robustness and generalization in complex, long-horizon tasks compared to strong baselines. 2 Related Work Figure 2: Illustration of the Trajectory-Aware Reward Shaping mechanism. The process consists of three stages: (1) Raw Experiences: Failed trajectories usually provide only sparse terminal feedback, obscuring where the execution first goes wrong. (2) Failure-point Detection: We identify the first breakdown step where the rollout deviates from valid execution according to per-step validity scores. (3) Prefix Credit Assignment: We assign positive rewards only to valid steps before the breakdown, while penalizing all invalid steps along the trajectory, producing dense and stable training signals for long-horizon optimization. 2.1 Vision-based GUI Agents The evolution of GUI agents has shifted from dependency on structured metadata (e.g., DOM trees, View Hierarchies) to purely vision-based approaches that leverage MLLMs. Early works like CogAgent Hong et al. 2024 introduced high-resolution visual encoders to handle small GUI elements. Recently, the field has moved towards generalist agents capable of cross-platform operation. Qin et al. 2025 proposed UI-TARS, an end-to-end native GUI agent that achieves SOTA performance on benchmarks like OSWorld ( Xie et al. 2024 ) by unifying perception and action spaces. However, while these models excel at atomic grounding, they often struggle with tasks requiring long-term planning and error correction. A promising trend is the transition from reactive actors to deliberative reasoners. InfiGUI-R1 Liu et al. 2025 incorporates a "System 2" reasoning stage before action execution, significantly reducing hallucination in complex workflows. Unlike these methods that primarily focus on SFT, our work explores how to further optimize these vision-centric agents through reinforcement learning to handle long-horizon dependencies. 2.2 Reinforcement Learning for GUI Automation While SFT provides a solid initialization, it struggles to generalize to unseen states due to the discrepancy between the expert’s policy and the agent’s learned policy during inference. RL has been adopted to mitigate this. DigiRL Bai et al. 2024 pioneered an offline-to-online curriculum but faces high interaction costs in real-time environments. To address