Domain Adaptation with Adaptive Imagination for Visual Reinforcement Learning under Limited Target Data

Hyunwoo Park, Sang-Hyun Lee · 2026

Read Domain Adaptation with Adaptive Imagination for Visual Reinforcement Learning under Limited Target Data by Hyunwoo Park, Sang-Hyun Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30192.txt.

Introduction

Hyunwoo Park and Sang-Hyun Lee help robot policies trained in simulation survive contact with reality. Vision-based reinforcement learning suffers when real images differ from simulated ones. Domain adaptation can bridge that gap, but it usually needs much target data. Real target data is exactly what practice rarely provides. Their framework, AIDA, works under severe target data scarcity.

AIDA's core idea is adaptive imagination for augmenting the little data available. It generates imagined rollouts, but only trusts the reliable ones. A distribution-shift-aware discriminator cuts rollouts that drift into low-confidence regions. On the trusted transitions, a self-consistency loss adds training signal. That loss cycles state to image and back to state. Representations become semantically meaningful rather than superficially matched. Experiments cover five MuJoCo tasks and two robotics tasks. AIDA outperforms the baselines across that whole suite.

Catalog record

CREATORHyunwoo Park, Sang-Hyun Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30192

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Domain Adaptation with Adaptive Imagination for Visual Reinforcement Learning under Limited Target Data Hyunwoo Park, Sang-Hyun Lee Abstract Sim-to-real transfer remains a major obstacle for reinforcement learning (RL), especially for vision-based control where image observations exacerbate the state-distribution shift between simulation and the real world. Domain adaptation (DA) is a promising remedy for this challenge. Prior sim-to-real DA works have demonstrated encouraging results, yet these approaches typically assume substantially more target data, which is not available in practice. Indeed, their performance degrades significantly when the target data budget is reduced. To address this challenge, we propose AIDA ( A daptive I magination for D omain A daptation), a domain adaptation framework for visual reinforcement learning that addresses sim-to-real transfer under scarce target data without requiring additional interaction with the target environment. Our key idea is adaptive imagination: generating reliable and semantic imagination rollouts to augment limited target data. Specifically, AIDA employs a distribution-shift-aware discriminator that truncates rollouts when imagined transitions drift into low-confidence regions, so that only reliable transitions contribute to the augmentation. On these reliable transitions, AIDA introduces a self-consistency loss that cycles through state → image observation → state, penalizing discrepancies between the original and reconstructed states. This provides additional adaptation signals beyond the scarce target data. Our experiments demonstrate that adaptive imagination effectively truncates unreliable rollouts. By enforcing a self-consistency loss on the resulting reliable transitions, AIDA learns semantically meaningful state representations and outperforms baselines across five MuJoCo tasks and two Gymnasium-Robotics tasks. 1 Introduction Deep reinforcement learning (RL) has achieved remarkable success in simulation, enabling agents to master complex control tasks from raw sensory inputs [ 12 , 1 ] . However, deploying these learned policies in the real world remains a fundamental challenge: discrepancies between simulation and reality—known as the sim-to-real gap—often cause dramatic performance degradation [ 24 ] . This gap can be broadly attributed to two factors: dynamics mismatch and state-distribution shift. In particular, for vision-based RL policies, state-distribution shift often plays a larger role. This is because pixel observations are high-dimensional and highly sensitive to changes in appearance and sensing conditions, which exacerbates the sim-to-real gap. In this work, we focus on mitigating this state-distribution shift for vision-based RL policies. To cope with such distribution shifts, three families of methods are commonly used: domain randomization (DR), domain generalization (DG), and domain adaptation (DA). DR [ 18 , 9 , 15 , 10 ] augments the simulator to train a single robust policy, hoping that the resulting variability covers factors of variation in the target environment. However, when the test domain is truly unknown or the shift is large, such coverage cannot be guaranteed. DG [ 3 , 21 , 7 ] aims to learn policies that transfer to previously unseen test environments without target-domain supervision, typically by encouraging domain-agnostic representations and invariances (e.g., via attention/invariance objectives or augmentation-consistent regularization) rather than explicitly matching to a specific target domain. However, because the test domain is unknown by design, there is no guarantee that the training sources or augmentations capture the true factors of variation encountered at deployment; consequently, when the domain gap is large or includes unmodeled shifts, DG performance may degrade substantially—revealing a fundamental limitation of methods that must anticipate the target distribution without ever observing it [ 21 ] . DA [ 6 , 20 , 2 , 16 ] is a promising alternative that directly leverages target-domain data to adapt to the specific target environment. This requires access to target data, but in return enables specialization to the particular deployment setting, even when the underlying causes of the shift are unknown at training time. Prior DA methods have indeed demonstrated superior performance over DR and DG [ 20 , 6 ] . Nevertheless, they typically assume access to sufficient target data, and their performance degrades significantly when this budget is reduced [ 2 , 13 ] . Moreover, most prior DA methods adopt an image-to-image setting [ 6 , 20 , 16 ] , which requires training RL agents in image-based simulation—a process that is usually harder [ 11 , 23 ] and can slow down RL by up to 20 \times [ 22 ] . These two practical constraints remain underexplored despite their practical relevance. To address these challenges, we propose AIDA ( A daptive I magination for D omain A daptation), a DA framework for visual RL that operates under scarce target data without additional target interaction. Our key idea is adaptive imagination : augmenting limited target data with reliable imagination rollouts obtained via discriminator-gated truncation, and optimizing a self-consistency objective on these rollouts to learn semantically meaningful representations. Specifically, AIDA adopts a cross-modality setting (sim-state \rightarrow real-image) that avoids the overhead of image-based RL training. A learned dynamics model produces policy-conditioned rollouts starting from target-inferred states, augmenting the limited target data with synthetic transitions. However, longer rollouts accumulate model error and drift from the target manifold. Therefore, a three-way discriminator assesses each imagined transition and truncates the rollout when confidence drops, so that only reliable, target-aligned transitions contribute to training. This adaptive truncation is critical, as imagination reliability varies across both tasks and individual states, making any fixed imagination horizon suboptimal. On the resulting reliable transitions, we introduce a self-consistency loss that enforces a state \rightarrow image \rightarrow state cycle on discriminator-gated imagination rollouts. Unlike prior cycle objectives limited to short, data-supported predictions, our self-consistency loss extends cycle supervision to multi-step, policy-conditioned imagination rollouts, extracting additional training signal specifically from states the agent is likely to visit and thereby promoting semantically meaningful state representations. We evaluate AIDA across five MuJoCo tasks and two Gymnasium-Robotics tasks with scarce pre-collected target data, where it consistently outperforms baselines, demonstrating its effectiveness in practically important yet challenging regime. Our contributions are: (i) AIDA A novel paradigm that augments limited target data with reliable synthetic transitions and leverages self-consistency to learn semantically meaningful representations. (ii) Discriminator-gated imagination. A mechanism for generating only reliable imagined transitions by dynamically truncating imagination rollouts based on the discriminator’s domain classification signal. (iii) Self-consistency via imagination-augmented data. A scheme that learns semantically meaningful representations by enforcing a self-consistency loss on synthetic policy-conditioned imagination rollouts, providing additional supervision under limited target data. (iv) Experimental evaluation. Comprehensive experiments across five MuJoCo tasks and two Gymnasium-Robotics tasks demonstrate that imagined transitions progressively become unreliable as the rollout horizon grows and that discriminator-gated truncation yields higher returns than any fixed-horizon alternative, enabling AIDA to consistently outperform baseline DA methods in the low-data regime. 2 Related Work Bridging the sim-to-real gap has been a longstanding challenge in RL, and a number of approaches have been proposed to address the shift in state distributions between source and target domains. Broadly, these methods fall into three categories: domain randomization (DR), domain generalization (DG), and domain adaptation (DA). Domain Randomization (DR). DR trains an agent in simulation under a wide range of environment parameters—such as textures, lighting, camera poses, and object appearances—with the goal that such visual variations will cover the target-domain state distribution. In vision-based settings, this typically involves randomizing camera poses, lighting conditions, object placements, textures (including non-realistic ones), and backgrounds [ 18 , 9 , 15 , 10 ] . The resulting

Read the complete text