DADP: Domain Adaptive Diffusion Policy Pengcheng Wang, Qinghang Liu, Haotian Lin, Yiheng Li, Guojian Zhan, Masayoshi Tomizuka, Yixiao Wang Abstract Learning domain adaptive policies that can generalize to unseen transition dynamics, remains a fundamental challenge in learning-based control. Substantial progress has been made through domain representation learning to capture domain-specific information, thus enabling domain-aware decision making. We analyze the process of learning domain representations through dynamical prediction and find that selecting contexts adjacent to the current step causes the learned representations to entangle static domain information with varying dynamical properties. Such mixture can confuse the conditioned policy, thereby constraining zero-shot adaptation. To tackle the challenge, we propose DADP ( D omain- A daptive D iffusion P olicy), which achieves robust adaptation through unsupervised disentanglement and domain-aware diffusion injection. First, we introduce Lagged Context Dynamical Prediction, a strategy that conditions future state estimation on a historical offset context; by increasing this temporal gap, we unsupervisedly disentangle static domain representations by filtering out transient properties. Second, we integrate the learned domain representations directly into the generative process by biasing the prior distribution and reformulating the diffusion target. Extensive experiments on challenging benchmarks across locomotion and manipulation demonstrate the superior performance, and the generalizability of DADP over prior methods. More visualization results are available on the website . † † affiliationnotice: ∗ equal contribution, order is decided randomly 1 Introduction Learning-based polices have achieved remarkable success recently, enabling agents to solve increasingly complex decision-making problems ( 19 ; 28 ) . Despite these advances, most existing approaches remain coupled to a specific environment or operating condition ( 31 ; 1 ) , and their performance often degrades when deployed in the unseen domains ( 8 ) , which limits the practical applicability of learning-based policies . This mismatch highlights a fundamental challenge: designing a single policy that can generalize efficiently and robustly across domains remains critically important but inherently difficult. Figure 1: Averaged Normalized Performance of baselines across In-Distribution and Out-of-Distribution settings across all tasks. The results are normalized with random and expert policy performance. Most prior approaches begin by extracting domain information and leveraging it for decision-making, namely domain representation learning and representation utilization. Regarding the former, many methods extract representations through contrastive learning ( 35 ; 33 ) , whose performance often depends on carefully designed objectives and extra data generation. Some other approaches instead employ dynamical prediction as an auxiliary task to implicitly learn the domain representations from transitional dynamics ( 16 ; 6 ) ; however, the resulting representations are frequently of limited quality due to entangling the static domain information with varying dynamical properties. Regarding the latter, most methods utilize representations through input concatenation ( 13 ; 14 ) ,namely using representations as extra network input, or rely on sequence-modeling architectures ( 32 ; 25 ; 11 ) to capture domain information in an implicit, end-to-end manner. Such approaches often fail to fully leverage the learned representations, resulting degraded performance. To tackle these challenges, we propose DADP ( D omain- A daptive D iffusion P olicy), which achieves robust adaptation through unsupervised disentanglement and domain-aware diffusion injection. First, to remove disentangle time-varying properties from the unsupervisedly learned representation, we introduce Lagged Context Dynamical Prediction. Specifically, we break the temporal correlation between the context and the current step by introducing a large historical offset \Delta t\rightarrow\infty , preventing time-varying information in the context from assisting dynamical prediction and thereby excluding it from the extracted representation during learning. Second, we inject the learned representations directly into the generative process. Specifically, we start denoising with a representation-biased mixed guassian distribution, and reformulate the diffusion target to include the learned representation. We evaluate DADP across locomotion and manipulation tasks across MuJoCo and Adroit, showing consistently superior performance in both modalities capturing ability and generalizability. In summary, our main contributions are as follows: • Unsupervised Representation Learning. We propose Lagged Context Dynamical Prediction, a simple yet effective approach for unsupervisedly learning domain representations from dynamical prediction. • Denoising with Representation-Prediction. Instead of conditioning, we utilize the domain representations by biasing the prior distribution and reformulating the diffusion target, enabling better policy performance. • Superior Performance. We evaluate DADP on a broader and more challenging set of domain adaption tasks, demonstrating consistently superior performance in domain-adaptivity under the zero-shot setting. • Open-sourced Dataset and Pipeline. We release a complete open-sourced codebase, including the algorithm, datasets, and data generation pipeline, allowing the community to easily customize our framework. 2 Related Work 2.1 Domain Adaptive Policy Sequential Modeling Policy. Regarding policy architecture, early efforts extend the observation to context composed of multiple consecutive state ( 13 ) to provide sufficient information for domain adapation. Subsequent works leverage mature sequence-to-sequence models to better utilize the contextual information, such as Transformers ( 2 ; 32 ) or Mamba ( 25 ; 11 ) . Among them, Locoformer ( 19 ) employs Transformer-XL ( 4 ) to enable information sharing across episodes and online improvement. However, these methods often rely on purely end-to-end learning, where the absence of intermediate supervision prevents the models from effectively exploiting the implicit dynamical information ( 4 ) . Meta RL and In-Context RL. Regarding algorithms, In Context Reinforcement Learning (ICRL) ( 15 ) and Meta Reinforcement Learning ( 5 ) methods constitute a widely adopted approach, designed to operate over a MDP set by learning from task-level variations during training. In-context Q Learning ( 18 ) feed the task representation into a causal transformer with value and policy head for efficient learning across domains. This reflects the dominant approach adopted by most prior works ( 13 ; 35 ) on ICRL, where the learned representations are provided as additional input to the network to enable domain-aware decision making. Closest to our work, MetaDiffuser ( 23 ) also incoperates learned domain representations into the diffusion process. However, its representations suffer from the entangled time-varying information due to its representation learning pipeline, which can be solved with our proposed Lagged Context Dynamical Prediction. 2.2 Domain Representation Learning. Domain refers to the environment’s transition dynamics. In practice, it is often characterized by a low-dimensional parameter vector, including environmental parameters (e.g., gravity, friction) or agent-specific parameters (e.g., joint torques, limb lengths). To address the problem of domain adaptive policy learning, many prior works focused on domain representation learning, where a compact representation of domain is inferred from a trajectory of interactions. Supervised. Some works adopt a supervised learning setting, assuming that each domain can be characterized by a low-dimentional accessible environmental factor ( 36 ; 22 ) , which naturally serves as the target for representation learning. One of the most well-known works is RMA ( 13 ; 14 ) , which achieves online adaptation to different environments by co-training a factor-supervised context encoder and an representation-conditioned policy. Unsupervised. Many other prior works focus on the unsupervised setting, where such environmental factors are assumed to be unavailable. One line of work leverages classical unsupervised learning techniques to cluster data of different domains, such as contrastive learning–based approaches ( 17 ; 30 ) . Among them, CORRO ( 35 ) proposes a contrastive learning framework for robust task representations under distribution shifts between training and test be-havior policies. However, such methods often rely on, yet fail to fully exploit, the temporal and sequential structure inherent in control tasks, and are highly dependent on the quality of the extra data generation model. In contrast, our approach is derived from dynamics prediction formulated through sequence modeling, enabling a simpler and more effective capture of domain information. Another line of work learn domain representations implicitly by introducing dynamics prediction as an auxiliary task, like CaDM ( 16 ) , IIDA ( 6 ) and CARoL ( 10 ) . However, such methods often suffer from poor representation quality, as they also fail to properly remove the varying information present in the context. In contrast, our method breaks such time-local cues by reconstructing prediction pairs, thereby yielding representations that serve as domain-specific static representations. 3 Preliminaries 3.1 Problem Formulation In this work, we formulate the domain adaptive policy learning as an offline meta-RL problem. Specifically, we consider a task set \mathcal{T}=\{\mathcal{T}_{i}\}^{n}_{i=1} , where each task \mathcal{T}_{i} consists of an Markov Decision Process (MDP) and a policy that has been pre-trained on this MDP. \mathcal{T}=\{\mathcal{T}_{i}\}^{n}_{i=1}=\{\left(\mathcal{M}_{i},\pi_{i}\right)\}^{n}_{i=1} (1) The MDP can be defined by a tuple \mathcal{M}_{i}=(\mathcal{S},\mathcal{A},R,p_{i}) , where \mathcal{S}\subseteq\mathcal{R}^{n} is a continuous state space, \mathcal{U}\subseteq\mathcal{R}^{m} is a continuous action space, R:\mathcal{S}\times\mathcal{U}\rightarrow\mathbb{R} is the reward function, and p_{i}:\mathcal{S}\times\mathcal{U}\times\mathcal{S}\rightarrow[0,\infty) is the transition probability function. Across all MDPs, the state space \mathcal{S} , action space \mathcal{A} , and reward function R are shared, while the transition dynamics p_{i} differ across tasks, i.e., p_{i}\neq p_{j} . For each task \mathcal{T}_{i} , the agent is given an offline dataset \mathcal{D}_{i} , collected by executing a domain-specific expert policy \pi_{i} in the corresponding environment \mathcal{M}_{i} . The expert policy \pi_{i} is constructed by training a reinforcement learning (RL) agent to (near-)optimality on \mathcal{T}_{i} , and is therefore specialized to the dynamics and reward of \mathcal{M}_{i} . Our objective is to learn a policy \pi using the datasets \{\mathcal{D}_{i}\} from the training task set \mathcal{T}^{\mathrm{train}} , and to maximize discounted return for all tasks in \mathcal{T}^{\mathrm{train}}\cup\mathcal{T}^{\mathrm{test}} , i.e., J(\pi)\;=\;\mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty}\gamma^{t}\,R(s_{t},a_{t},s_{t+1})\right], where \gamma\in(0,1) is the discount factor. 3.2 Diffusion Policy Diffusion policies ( 3 ; 9 ) model the action generation as a stochastic denoising process conditioned on the observatoin. Specifically, a diffusion policy learns a conditional action distribution q(a|s) through a predefined forward diffusion process and a learned reverse denoising process. The forward process gradually perturbs a clean action a^{0} into noisy latent variables a^{t} by q(a^{t}\mid a^{t-1})=\mathcal{N}\bigl(a^{t}\mid\sqrt{\bar{\alpha}_{t}}\,a^{t-1},\Sigma_{t}\bigr), (2) where \mathcal{N}(\mu,\Sigma) denotes a Gaussian distribution with mean \mu and variance \Sigma_{t} , \bar{\alpha}_{t}\in\mathbb{R} is the variance schedule, and a^{0}\sim q(a|s) is an action sampled from the data distribution. Starting from Gaussian noise, actions are generated by iteratively applying the learned reverse process. The denoising policy \epsilon_{\theta}(a_{t},s,t) predicts the noise at each diffusion step conditioned on input. The policy is trained by minimizing a simplified surrogate objective ( 9 ) : \mathcal{L}_{\text{diff}}(\theta)=\mathbb{E}_{a^{0},\,\epsilon,\,k}\bigl[\lVert\epsilon-\epsilon_{\theta}(a^{k},s,k)\rVert^{2}\bigr], (3) where a^{t}=\sqrt{\bar{\alpha}_{t}}\,a^{0}+\sqrt{1-\bar{\alpha}_{t}}\,\epsilon,\epsilon\sim\mathcal{N}(0,I). The objective encourages the model to recover the injected noise at each diffusion step. At inference time, the policy samples an action by initializing from Gaussian noise and iteratively applying the learned denoising model conditioned on the current state to get the clean action distribution. 4 Domain Adaptive Diffusion Policy 4.1 Learn Representation by Extracting Static Info To enable the policy to possess domain adaptive capability, we firstly train a context encoder E_{\phi}(\cdot) to learn an effective domain representation z_{t} from the context \tau_{t} . We choose to learn the representation from context since (i) the domain factors that govern dynamics are typically latent and must be inferred from observed transitions, and (ii) at test time, the agent generally has access only to interaction history rather than privileged environment parameters. In this work, we learn the representation implicilty from dynamical prediction. Typically, the context as encoder input is from the most recent history ( 23 ) : \displaystyle z_{t}=E_{\phi}(\tau_{t});\hat{s}_{t+1}=f_{\theta}(s_{t},a_{t},z_{t}); (4) \displaystyle\tau_{t}=(s_{t-H},a_{t-H},\ldots,s_{t-1},a_{t-1}). This context selection is intuitive, as it aligns with the usage in online policy inference, where the most recent history is used as context. Note that there are two types of necessary information that can be infered from the context, which are both necessary for accurate next state prediction: static information \xi that represents the domain-specific dynamics (e.g. gravity), varying information \omega_{t} that includes instantaneous dynamical properties not captured in the current state (e.g. higher-order temporal derivatives of states): \displaystyle s_{t+1}=f(s_{t},a_{t},{\color[rgb]{0,0,1}\xi},{\color[rgb]{1,0,0}\omega_{t}});\ z_{t}=E_{\phi}(\tau_{t})=({\color[rgb]{0,0,1}\xi^{z}},{\color[rgb]{1,0,0}\omega_{t}^{z}}), (5) where f represents the ground-truth forward dynamics, {\color[rgb]{0,0,1}\xi^{z}} and {\color[rgb]{1,0,0}\omega_{t}^{z}} represent the inferred information from the context. Note that since the context is drawn from the most recent history, the inferred varying information {\color[rgb]{1,0,0}\omega_{t}^{z}} is temporally aligned with the ground-truth varying information {\color[rgb]{1,0,0}\omega_{t}} required for prediction task: \displaystyle z_{t} \displaystyle=\arg\min_{z=(\omega_{t}^{z},\xi^{z})}\mathbb{E}_{\mathcal{D}}\lVert s_{t+1}-\hat{s_{t+1}}\rVert^{2}. (6) As a result, z_{t}=({\color[rgb]{0,0,1}\xi},{\color[rgb]{1,0,0}\omega_{t}}) becomes a natural global minimum for the dynamical prediction task. However, recall that a domain corresponds to the environment’s transition dynamics, usually parameterized by a low-dimensional vector and therefore inherently static. For domain adaptation, the primary purpose of the representation is to provide a stable descriptor of the domain: it should reflect the persistent domain factors \xi across time, while remaining insensitive to ephemeral variations \omega_{t} that are not stable within a domain. Encoding \omega_{t} into z_{t} can cause representation drift within the same domain, reducing separability across domains and harming generalization when z_{t} is used as a domain descriptor for downstream policy learning. This motivates us to seek a mechanism for disentangling time-varying information \omega_{t} from z_{t} . To remove \omega_{t} , we propose Lagged Context Dynamical Prediction. Specifically, we introduce a temporal offset \Delta t to weaken the contribution of time-local cues in the context for next state prediction. With \Delta t , we can adjust the ”distance” between the context and the current timestep: \displaystyle \displaystyle z_{t-\Delta t}=E_{\phi}(\tau);\hat{s}_{t+1}=f_{\theta}(s_{t},a_{t},z_{t-\Delta t}); (7) \displaystyle\tau_{t-\Delta t}=(s_{t-H+1-\Delta t},a_{t-H+1-\Delta t},\ldots,s_{t-\Delta t},a_{t-\Delta t}), , where the original context corresponds to \Delta t=1 . From an information-theoretic viewpoint, as \Delta t increases, the offset context becomes less informative about instantaneous variations. Since z_{t-\Delta t} is a function of \tau_{t-\Delta t} , we have I({\color[rgb]{1,0,0}\omega_{t}};z_{t-\Delta t}\mid s_{t},a_{t},{\color[rgb]{0,0,1}\xi})\leq I({\color[rgb]{1,0,0}\omega_{t}};\tau_{t-\Delta t}\mid s_{t},a_{t},{\color[rgb]{0,0,1}\xi}) . In this way, z_{t-\Delta t} discards the instantaneous variations , while \xi remains informative since it is time-invariant within a domain. Consequently, optimizing prediction with offset contexts biases the representation toward the static domain factors \xi rather than transient \omega_{t} . When \Delta t\rightarrow\infty , the representation z_{-\infty}=({\color[rgb]{0,0,1}\xi},{\color[rgb]{0,0,1}\overline{\omega}}) becomes static, where {\color[rgb]{0,0,1}\xi} is the static domain information, {\color[rgb]{0,0,1}\overline{\omega}} is a averaged varying properties that minimizes the prediction loss on the dataset distribution. This can be easily achieved by selecting the context from another episode in the same domain. Please refer to Appendix A for a toycase explanation. Figure 2: t-SNE Visualization of Denoising Process of Standard Diffusion and DADP. The sampled points from prior distribution and utilized representation are contructed from the training datasets and learned context encoder. 4.2 Utilize Representation by Diffusion Modulation After obtaining high-quality domain representations, it remains to determine how to better utilize them to enable domain-aware decision-making. In this work, we build our method upon diffusion policy, as it has been widely adopted and has demonstrated strong performance across various control tasks. In the standard Diffusion Policy ( 3 ) , the denoising process starts from pure Gaussian noise, where different denoising trajectories are governed by single representation-conditioned policy: a^{t}=\sqrt{\bar{\alpha}_{t}}a^{0}+\sqrt{1-\bar{\alpha}_{t}}{\epsilon}. (8) If we simply take the learned representation as extra policy inputs, the diffusion policy has to reconstruct different domain-specific action modalities from every sampled point in the prior gaussian distribution equally. This entanglement leads to mixed denoising trajectories in the latent noise space, preventing the policy from exploiting the structure of the noise to better leverage domain information, and consequently resulting in degraded performance. To solve this challenge, intead of utilizing the representation as condition, we inject the representation to the generation. Specifically, DADP initializes the denoising process from a Gaussian Mixture by incorporating the learned representation z into the forward process. Following the formulation for structured diffusion models (e.g. Mixed DDIM ( 12 ) ), we define the perturbed state x_{t} as: a^{t}=\sqrt{\bar{\alpha}_{t}}(a^{0}-{z})+{z}+\sqrt{1-\bar{\alpha}_{t}}{\epsilon}, (9) where the fully noised data point at step T follows x_{T}=z+\epsilon , and the prior distribution is a mixed Gaussian distribution with multiple peaks, where each peak represents a domain-specific action modality. In this way, we inject the domain information into prior distribution as shown in Figure 2 . By rearranging Eq. ( 9 ), the clean action a^{0} can be estimated from a^{t} and the predicted noise \epsilon^{\theta} : a^{0}=\frac{a^{t}-{z}-\sqrt{1-\bar{\alpha}_{t}}{\epsilon}^{\theta}}{\sqrt{\bar{\alpha}_{t}}}+{z}. (10) Utilizing the Denoising Diffusion Implicit Model (DDIM) formulation ( 27 ) and omitting the stochastic noise injection for clarity, the reverse step is defined as: a^{t-1}=\sqrt{\bar{\alpha}_{t-1}}a^{0}+\sqrt{1-\bar{\alpha}_{t-1}}\frac{a^{t}-\sqrt{\bar{\alpha}_{t}}a^{0}}{\sqrt{1-\bar{\alpha}_{t}}}. (11) Substituting Eq. ( 10 ) into Eq. ( 11 ) yields: \begin{split}a^{t-1}=&\sqrt{\frac{\bar{\alpha}_{t-1}}{\bar{\alpha}_{t}}}\left(a^{t}-{z}-\sqrt{1-\bar{\alpha}_{t}}{\epsilon}^{\theta}\right)+\sqrt{\bar{\alpha}_{t-1}}{z}\\ +&\frac{\sqrt{1-\bar{\alpha}_{t-1}}}{\sqrt{1-\bar{\alpha}_{t}}}\left({z}+\sqrt{1-\bar{\alpha}_{t}}{\epsilon}^{\theta}-\sqrt{\bar{\alpha}_{t}}{z}\right).\end{split} (12) In this work, instead of setting {\epsilon}^{\theta} as the prediction target as usual, we propose a joint prediction objective, where the model learns to predict a composite term \hat{\epsilon}^{\theta} representing the noise and the representation shift together: a^{t}=\sqrt{\bar{\alpha}_{t}}a^{0}+\underbrace{\left(1-\sqrt{\bar{\alpha}_{t}}\right){z}+\sqrt{1-\bar{\alpha}_{t}}{\epsilon}}_{\hat{{\epsilon}}^{\theta}}. (13) Under this scheme, the sampling iteration simplifies as: a^{t-1}=\sqrt{\frac{\bar{\alpha}_{t-1}}{\bar{\alpha}_{t}}}\left(a^{t}-\hat{{\epsilon}}^{\theta}\right)+\frac{\sqrt{1-\bar{\alpha}_{t-1}}}{\sqrt{1-\bar{\alpha}_{t}}}\hat{{\epsilon}}^{\theta} (14) In this way, we not only bias the prior distribution, but also introduce extra supervision on each denoising steps to further guide and simplify the denoising process. A complete empirical analysis of these variants can be found in Section 4 , which shows the great policy performance gain of the proposed approach. 5 Experiments With experiments, we aim to answer these questions: 1. How does the performance of proposed DADP policy compared to existing SOTA methods? 2. Does the proposed Lagged Context Dynamical Prediction contributes to the representation quality? 3. Does the proposed representation utilization further improve the performance of the diffusion policy? 5.1 Experimental Setup Environments. Previous evaluations of domain adaptation policies have largely focused on existing locomotion settings ( 29 ; 23 ) , where domain randomization typically is restricted to mild variations (e.g., friction or gravity shifts), which tend to have limited impact on the optimal gait. In this work, we expand the locomotion tasks to four environments and further introduce morphological variations. As a result, the gaits across different domains exhibit greater diversity compared to prior works. Please refer to Appendix 7 for the dataset visualization. Furthermore, to demonstrate the generality and applicability of DADP in environments with complex dynamics, we additionally incorporate a manipulation benchmark, Adroit ( 26 ) , into our experiments. Data Generation. For locomotion environments, we follow the data collection pipeline of CORRO ( 35 ) , constructing the task set by sampling different environmental factors in the parametric space. For each task, we use SAC ( 7 ) to train a task-specifc expert for offline data collection, which contains 25 domains. For manipulation tasks in Adroit, we adopt the pre-collected dataset from ODRL ( 21 ) , which contains 3 domains. Please refer to Appendix B.1 for more details. Table 1: Benchmark performance across different environments. The results are from the last checkpoint, which are presented in mean \pm std across 5 random seeds, where the highest mean performance of each variant is bolded, and the second highest is underlined. Environment Setting Expert CORRO Prompt-DT Meta-DT DADP (Ours) HalfCheetah IID 4575 -301 \pm 42 1640 \pm 194 3857 \pm 234 4100 \pm 85 OOD – -246 \pm 36 250 \pm 375 3174 \pm 501 3056 \pm 113 Walker2d IID 7101 61 \pm 49 590 \pm 57 1304 \pm 586 3991 \pm 24 OOD – 66 \pm 69 435 \pm 157 889 \pm 579 3015 \pm 207 Ant IID 3598 -867 \pm 430 700 \pm 189 3045 \pm 128 3117 \pm 88 OOD – -962 \pm 553 208 \pm 126 3187 \pm 899 3495 \pm 102 Hopper IID 1555 80 \pm 11 935 \pm 65 1140 \pm 156 1643 \pm 37 OOD – 61 \pm 21 1148 \pm 150 1208 \pm 99 1711 \pm 60 Door IID 3233 -50 \pm 13 2116 \pm 177 1283 \pm 323 1483 \pm 55 OOD 3261 -58 \pm 2 1080 \pm 209 1294 \pm 228 1430 \pm 99 Relocate IID -1.92 -12.3 \pm 1.92 -7.44 \pm 0.10 -6.06 \pm 0.40 -5.57 \pm 0.24 OOD -1.70 -12.0 \pm 0.72 -6.47 \pm 0.36 -5.77 \pm 0.42 -5.63 \pm 0.21 Baselines. We consider the following methods as our baselines. Please refer to Appendix B.2 for more details. • CORRO ( 35 ) proposes a contrastive learning framework for robust task representations under distribution shifts, outperforming prior context-conditioned policy-based methods. • Prompt-DT ( 34 ) leverages Transformer-based sequence modeling with a prompt formulation to enable few-shot adaptation in offline RL, serving as a strong end-to-end meta-RL baseline. • Meta-DT ( 32 ) incorporates an additional learned domain representation as an augmented observation, further improving performance and representing a SOTA baseline in domain adaptation task. Training and Evaluation. Training is conducted in two stages. Firstly, a context encoder is pre-trained on training dataset to extract domain representations from trajectories; secondly, a diffusion policy is trained with the fixed learned context encoder across 5 random seeds. During evaluation, we test the policies with zero-shot setting, where the contexts are online collected during policy rollout. Compared to the few-shot setting, which assumes access to expert datasets from unseen domains as context, the zero-shot setting more closely reflects practical deployment scenarios ( 19 ) . We evaluate all the baselines under both In-Distribution (IID) and Out-of-Distribution (OOD) settings. The IID setting measures performance on all the domains present in the training dataset, assessing the policy’s ability to master multiple domains. For the OOD setting, we randomly sample environmental parameters from the factor space to construct 5 new domains, evaluating the policy’s generalizability to unseen domains. For the Adroit benchmark, we instead use the Easy and Hard domains for IID setting, and the Medium domain as the OOD setting. Please refer to Appendix E for more details. 5.2 Experimental Results As shown in Table 1 , across all evaluated environments, DADP consistently achieves strong performance under both IID and OOD settings, outperforming or matching the best-performing baselines in nearly all cases. These results indicate that DADP effectively captures and leverages the domain information to improve performance across in-distribution and out-of-distribution domains. Moreover, across all environments, DADP achieve stable performance with smaller standard deviation value across different seeds, demonstrating its strong stability and practicability. Figure 3: t-SNE visualization of walker representations learned with different \Delta t . 5.3 Ablation Study Table 2: Normalized Representative Metrics of Learned Embeddings with different \Delta t . Environment Metrics \Delta t=1 \Delta t=4 \Delta t=16 \Delta t=32 \Delta t=\infty Supervised Walker2d Linear Probe Accuracy 27.9% 35.7% 48.5% 64.9% 99.3% 99.8% Reconstruction Loss 476.1 427.5 312.6 229.0 3.2 1.0 HalfCheetah Linear Probe Accuracy 68.6% 84.7% 96.7% 98.3% 99.9% 99.9% Reconstruction Loss 45.9 19.8 3.4 1.1 0.4 1.0 In the ablation study, we aim to examine how each proposed component of our diffusion policy contributes to the overall performance. We focus on the two tasks where DADP achieves the different level of gains over the other baselines in the main results, Walker2d and HalfCheetah. Please refer to Appedix C for additional experiments and analysis. 5.3.1 effect of \Delta t on representation qualities To evaluate the representation qualities, we apply the learned context encoder to encode the training dataset, obtaining the domain representation set. Specifically, we use the linear probe accuracy ( 24 ) and reconstruction loss to evaluate the representation qualities. For linear probe accuracy, we train a single-layer softmax linear classifier to predict the one-hot domain index corresponding to each representation; for reconstruction loss, we train a two-layer MLP to predict the exact dynamical parameter vectors. Furthermore, we treat the supervised representation as an upper bound on performance, as it is trained with access to ground-truth labels—specifically in our setting, the environment parameters and the corresponding one-hot domain indices. As shown in the Table 2 , as \Delta t increases, both metrics consistently improve, eventually comparable to that of supervised representations This indicates that larger \Delta t effectively breaks time-local cues, resulting in representations with stronger representative capacity for domain classification and encoding underlying dynamical parameters. Moreover, we can explane the larger performance gain on Walker2d compared with the HalfCheetah, since the former representation quality benefit much more from larger \Delta t . We also provide the t-SNE visualizations of the representations of Walker2d learned with different \Delta t in figure 3 . As \Delta t increases, the representations from different domains gradually distinctly cluster. Compared some previous methods ( 35 ; 17 ) based on contrastive learning, our approach can achieve great embedding qualities from a simple objective without extra data generation. Furthermore, we validate that the static representation can lead to better policy performance. As shown in Table. 4 , condional policy can benefit from the higher-quality representations, especially in the walker2d, whose representations quality improves more. With the proposed utilization, the performance gain can achieve comparable performance with Supervised baseline whose representation is trained with supervised learning as in Sec. 5.3.1 . We also validate that the performance gain can be consistently achieved across reward-changing environments and different datasets with ablation on Meta-DT. As the results shown in Table 3 , by introducing larger \Delta t to the representation in the Meta-DT, we observe consistent performance gains across all domain adaptive environments and the majority of reward-changing environments, demonstrating the generalizability of the proposed simple yet effective representation learning technique. Table 3: Meta-DT performance with different \Delta t . Environment \Delta t=1 \Delta t=32 Point-Robot -10.7 -11.6 Ant-Dir 368.9 391.7 Cheetah-Dir 542.4 554.2 Cheetah-Vel -100.1 -98.7 Hopper-Param 342.0 363.6 Walker-Param 397.4 399.8 5.3.2 effect of representation utilization Table 4: Ablation Results on Representation Utilization. Variants Options Walker2d HalfCheetah Representation Utilization IID OOD Mastery IID OOD Mastery End-to-End Diffusion Null. Null. 3722 2852 40 \% 3509 2496 80 \% Conditional Policy \Delta t=1 Cond. 2093 1617 0 \% 3740 2594 80 \% Better Representation \Delta t=\infty Cond. 3394 1813 28 \% 3603 2744 76 \% Mixed DDIM \Delta t=\infty w/o Predict 3356 1908 36 \% 3533 3012 84 \% DADP (Ours) \Delta t=\infty w/ Predict 3991 3015 44 \% 4100 3055 96 \% Expert Null. Null. 7101 - 100% 4575 - 100% Supervised Supervised w/ Predict 4014 2540 44% 3846 3152 88% In this section, we investigate how different representation utilization affect the resulting policy performance. Specifically, we mainly focues on the following utilizations: 1. Null.: We remove the representation in policy input and generation, serving as an end-to-end baseline. 2. Cond.: We utilize the representation as the extra input to the policy for conditional generation. 3. w/o Predict: We utilize the representation to bias the prior distribution to a mixed guassian. 4. w/ Predict: Upon w/o Predict, we further utilize the representation as part of policy prediction target. As shown in Table 4 , DADP achieve superior performance across all the variants. This indicates that our proposed utilization strategy maximizes the effectiveness of the learned high-quality representations. To validate the source of performance gain, we introudce a new metric, mastery , which is ratio of IID domains that policy achieve 60% of the expert policy performance. Across all the variants, DADP achieves the highest mastery, showing its strong capability to master diverse domains and locate the target manifold of the corresponding domain, resulting better mastery across the domains. In real-world application, higher mastery implies that a single policy can adapt to a broader range of embodiments and diverse environments. Please refer to Appedix 7 for visualization results. To visualize if the representation-prediction utilization can enable effective denoising process, we first rollout the corresponding variants in a specific domain. Next, we split the trajectories into contexts, apply the learned context encoder and visualize the representations of the resulting trajectories. As shown in Figure 4 , compared with Mixed DDIM and Conditional Policy, DADP representation accurately locate the running policy to the target domain thus better leverage the in-distribution capability for better control performance. Another interesting obeservation is, despite its simplicity, End-to-End Diffusion outperforms many variants. This suggests that diffusion-based policies constitute a particularly well-suited policy architecture for domain adaptation problems, and remain underexplored in this context. Figure 4: Walker Online Adaptation Representation Visualizations 6 Conclusion We propose DADP, a diffusion policy achieves robust domain adaptation through unsupervised disentanglement and domain-aware diffusion injection. To obtain high-quality domain representations unsupervisedly, we propose Lagged Context Dynamical Prediction to remove the time-varying information presents in the context. With the learned representations, we bias the prior distribution and reformulate the diffusion target, achieving SOTA performance and generalizability across diverse challenging benchmarks with verifiable analysis and visualization. Limitations. In this work, we focus on stationary (time-invariant) dynamics and therefore distangle static information from the time-varying information in the context. Nonetheless, time-varying signals can be crucial in non-stationary environments, where they may reflect evolving dynamics that a policy must track for effective control. In future work, we plan to explore how to jointly disentangle and retain the time-varying information, and extend DADP to non-stationary dynamical environments settings. Appendix A Toycase of \Delta t Design Figure 5: Intuition of the \Delta t desing: since the varying velocity inferred from another episode in the same domain can not assist the prediction, only static gravity will be extracted in the representation. The toycase can be illustrated with Figure 5 . We consider a toy example of a ball vertical projectile motion under gravity without air resistancem, whose time-step length t_{0} is known. At each time step, the state of the ball is represented solely by its vertical position s_{t}=y_{t} , which constitutes an incomplete state representation without the vertical speed v^{y}_{t} . Our goal is to infer the unique scalar environmental factor of this system—the gravitational acceleration g —by predicting the next state: y_{t+1}=y_{t}+v_{t}^{y}t_{0}+\frac{1}{2}gt_{0}^{2} (15) Consider the prediction with the most recent context with \Delta t=1 , \tau_{\Delta t=1}=\left(y_{t-3},y_{t-2},y_{t-1}\right) , which is encoded as z_{\Delta t=1}=E_{\phi}(\tau_{\Delta t=1}) . Note that for continuous states of length L=3 , there are always two types of information that can be extracted from the sequence \left(y_{T-2},y_{T-1},y_{T}\right) . • Static gravity {\color[rgb]{0,0,1}g}=\frac{1}{t^{2}_{0}}\left(y_{T}+y_{T-2}-2y_{T-1}\right) • Varying speed {\color[rgb]{1,0,0}v^{y}_{T}}=\frac{1}{2t_{0}}\left(4y_{T-1}-3y_{T-2}-y_{T}\right) Due to the time-local cues, the extracted velocity from the context can assist the prediction to achieve lower prediction loss by extend itself to the current step. v^{y}_{t}=v^{y}_{t-1}+gt_{0}=v^{y}_{T}+gt_{0} (16) We assume that the neural network can eventually achieve zero loss on this prediction; under this assumption, the learned representation must encode both types of information: z_{\Delta t=1}=\arg\min_{z=(g_{z},v_{z})}\mathbb{E}\lVert y_{t+1}-\hat{y_{t+1}}\rVert^{2}=\arg\min_{z=(g_{z},v_{z})}\mathbb{E}\lVert g-g_{z}\rVert^{2}+\lVert v^{y}_{t}-v_{z}\rVert^{2}=({\color[rgb]{0,0,1}g},{\color[rgb]{1,0,0}v^{y}_{t-1}}+{\color[rgb]{0,0,1}gt_{0}}) (17) However, this makes the representation a mixture of varying information and static domain representation, while only the latter is desired for representation learning. To remove the time-varying component, one can reduce the influence of time-local cues by simply increasing the \Delta t , namely the distance between the context and the current step. Now consider the context from another episode in the same domain \tau_{\Delta t=\infty}=\left(y_{t-3},y_{t-2},y_{t-1}\right) . In this case, the varying speed can not be extended to the current step. To predict next state with lowest loss, the encoder would be enforced to learn the static information: gravitational acceleration g and the averaged velocity \overline{v^{y}}=\mathbb{E}_{\mathcal{D}}\left[v^{y}_{t}\right] : \displaystyle z_{\Delta t=\infty} \displaystyle=\arg\min_{z=(g_{z},v_{z})}\mathbb{E}\lVert y_{t+1}-\hat{y_{t+1}}\rVert^{2}=\arg\min_{z=(g_{z},v_{z})}\mathbb{E}\lVert g-g_{z}\rVert^{2}+\lVert v^{y}_{t}-v_{z}\rVert^{2}=({\color[rgb]{0,0,1}g},{\color[rgb]{0,0,1}\overline{v^{y}}}) (18) In this case, we recover a static estimate of the gravitational acceleration. Meanwhile, the additionally learned average velocity term serves as complementary static information that characterizes the overall distribution, enabling the representation to remain stable within each domain while also capturing salient behavioral patterns. Appendix B Implementation Details B.1 Expert Dataset Dataset Generation. We generated datasets across four MuJoCo environments: Ant, HalfCheetah, Walker2d, and Hopper. For each environment, we varied specific physical parameters to create diverse dynamics. For Ant, HalfCheetah, and Hopper, the datasets comprise 25 distinct sets of dynamics parameters, with each set containing 100 episodes. Due to the higher complexity of the Walker2d environment, we generated 25 distinct dynamics parameters with 300 episodes per set to ensure sufficient coverage. The maximum episode length is set to 1,000 transitions. Due to its inherent instability of Hopper expert data generation, we limit its transitions to 500 and do not introduce morphological variations. To generate the data, we trained a Soft Actor-Critic (SAC) policy for each parameter setting and collected rollouts from the resulting policies. For the Adroit manipulation domain, we utilized datasets from ( 21 ) , specifically selecting the relocate-shrink-finger and door-shrink-finger tasks. Following the benchmark protocols, we used the ”Easy” and ”Hard” variants for training. Refer to ( 21 ) for the specific configurations of these tasks. Table 5: Dynamics parameter ranges for the MuJoCo environments. Environment Parameter Range Ant Leg Length (4 legs) [0.465,1.612] Hopper Joint Damping (3 joints) [0.75,1.19] Friction [0.75,1.19] HalfCheetah Back Leg Mass [2.99,6.69] Torso Length [0.336,0.951] Head Length [0.084,0.238] Front Leg Lengths [0.075,0.400] Walker2d Friction [1.32,2.28] Torso / Foot Length [0.30,0.67] Thigh / Leg Length [0.27,0.51] Mass (Left Thigh/Leg/Foot) [2.99,6.69] B.2 Baselines We benchmark our method against four baselines representing distinct training paradigms, including an offline RL-based approach, two Decision Transformer (DT)-based approaches, and one diffusion-based approach. For reproducibility, we utilize the official implementations for all baselines, with specific adaptations for our setting as detailed below: CORRO ( 35 ) . We employ the official implementation of this robust offline meta-RL method. It utilizes contrastive learning to acquire robust task representations, generating a latent representation from the task context. The offline RL policy is then conditioned on this representation to handle distribution shifts effectively. Prompt-DT ( 34 ) . Building on the official Decision Transformer implementation, this method utilizes trajectory segments as prompts to encode task information. While the standard inference pipeline samples prompts from an expert dataset, we modify this process to ensure a fair comparison under our zero-shot setting. Specifically, we construct the prompt using the agent’s recent interaction history directly gathered from the environment. Meta-DT ( 32 ) . We utilize the official Meta-DT codebase, which trains an encoder to compress trajectory segments into latent representations. These representations are processed by a world model (comprising a dynamics decoder and reward decoder). A Decision Transformer then conditions on this representation to predict future actions. In our experiments, as we focus exclusively on dynamics shifts, the reward decoder component is omitted. End-to-End Diffusion ( 20 ) . We adopt the modular architecture from the official implementation of DV ( 20 ) , which decomposes the diffusion policy into three distinct components: a planner, reward guidance, and an optional inverse dynamics policy. In our deployment, we utilize a Diffusion Transformer (DiT) as the backbone for the planner. To inject dynamics information, we condition the planner on the history trajectory and utilize the diffusion model to inpaint the future trajectory. Consistent with DADP, we exclude the reward guidance module. B.3 The details of DADP Context Encoder Architecture : The context encoder is implemented with a Transformer encoder with apative pooling at the output layer. Specifically, Two separate MLPs (state/ action encoder) map raw state and actions to the same dimension, and interleave the state tokens and action tokens to form a token sequence. Next, add learnable positional representations and apply dropout, and feed the position-augmented tokens into a Transformer encoder to produce output. Finally, aggregate the sequence using a learnable-query multi-head attention pooling to get a single vector as the representation. Diffuion Policy Architecture : The diffusion policy in DADP is adapted from the official implementation of DV. We modify the forward and denoising processes according to the formulations in ( 9 )–( 14 ). All other components, including model hyperparameters and network architecture, remain identical to the original DV implementation. The corresponding hyperparamters are shown in Table 6 . Table 6: Context Encoder Hyperparameters and Experimental Settings Hyperparameter Value Context Encoder Architecture Model Dimenstion 256 MLP Hidden Dimension 256 Feedforward Hidden Dimension 1024 Hidden Layer 4 Adaptive Pooling Heads 8 Adaptive Pooling Dropout 0.1 Attention Heads 8 History Length ( H ) 16 Task Representation Dim. \dim(s)+\dim(a) Context Encoder Training Batch Size 128 \beta_{\text{forward}} 1.0 \beta_{\text{inverse}} 1.0 Training Ratio 0.8 Learning Rate 3e-4 Epochs 10 Policy Architecture Hidden Dimension 256 Planner Depth 6 Attention Heads 8 History Length ( H ) 16 Prediction Horizon 4 Noise Schedule Cosine Policy Training Batch Size 256 Learning Rate 3e-4 MuJoCo Iterations 1e6 (Walker), 4e5 (Ant, Hopper), 1e5 (HalfCheetah) Adroit Iterations 5e5 (Relocation), 1e5 (Door) Inference & Evaluation Inference Steps 5 Guidance Scale 0.1 (Ant: 0.05) Max Env Steps 1000 (MuJoCo), 200 (Adroit) Eval. Episodes 50 (MuJoCo), 200 (Adroit) Appendix C Addtional Experiments C.1 Effect of different guidance scale In this section, we conduct an ablation study over a range of values for the introduced guidance scale. As shown in Table 7 , the results indicate that our method is not highly sensitive to this coefficient, with performance degrading sharply only when the guidance scale becomes excessively large. This demonstrates that our approach can achieve excellent performance without requiring extensive hyperparameter tuning. Table 7: Ablation Results under Different Guidance Scales. Environment Setting 0 0.01 0.05 0.1 0.5 1 5 Walker2d IID 3722 4026 4231 3957 3968 3615 2759 OOD 2852 2721 2837 2681 2934 2891 1854 HalfCheetah IID 3509 3920 3808 4079 4114 4093 404.5 OOD 2496 2931 2604 3021 2934 3162 39.5 C.2 Context Source In DADP, since the representation represents static domain information, its context source is not restricted to online-collected recent history. In this section, we further consider several practical deployment settings to substantiate the properties of the learned representations and the general applicability of DADP. We continue to assume that, in an unknown domain, no ground-truth policy rollouts are available, as this represents the most realistic scenario when deploying a policy to a new domain. Specifically, we consider the following three variants of context source: • Cold Start : DADP adopted approach, whose context is online collected recent history. When the online recent history length is insufficient, padding states and actions are used to complete the context window. • Persistent Context : By executing the Cold Start policy, we have the in-domain policy rollouts as context source. We randomly sample a clip in the policy rollouts as the persistent context used during online inference. • Warm Start : Following the Persistent Prompt, we replace the context source from policy rollouts to online recent history when length is sufficient. The context from policy rollouts is only used as warm start prompt. Table 8: Ablation Results on Context Source. The normalized metric is the averaged performance normalized with the Expert IID. Variants Walker2d HalfCheetah Hopper Ant Normalized IID OOD IID OOD IID OOD IID OOD IID OOD Cold Start 3985 2765 4079 3045 1692 1809 3069 3414 0.851 0.797 Persistent Context 3988 2938 4080 2902 1679 1828 3206 3552 0.858 0.809 Warm Start 4117 2833 4070 2846 1688 1662 3221 3670 0.865 0.783 We evaluate the different variants with the same checkpoint across the MuJoCo environments. As shown in Table 8 , different variants achieve comparable performance, further validating the static nature of the learned representations and the resulting flexibility in the choice of context sources. Moreover, this property enables the use of pre-collected rollouts obtained by executing the policy in the environment to mitigate the cold-start phase with incomplete context, thereby improving stability and performance once the context is fully populated. Appendix D Visualization D.1 Representation Visualization We also provide the t-SNE visualizations of the representations of HalfCheetah learned with different \Delta t in figure 6 . As \Delta t increases, the representations from different domains also become gradually distinctly clustered. We observe from the quantitative results that, in Walker2d, increasing \Delta t yields a substantially larger improvement in representation quality compared to HalfCheetah. This trend is also reflected in the visualization: when \Delta t=1 , different domains already exhibit partial clustering behavior, and for some domains, increasing \Delta t leads to improved cluster separation. As \Delta t\rightarrow\infty , the resulting representations achieve high quality comparable to those observed in the Walker2d setting. Figure 6: t-SNE visualization of half cheetah representations learned with different \Delta t . D.2 Domain-specific Action Modalities In this section, we provide visualizations of different domain-specific gaits presented in different tasks in Figure 7 , 8 , 9 , 10 . As mentioned in Appendix B.1 , we do not introduce morphological variations in the Hopper environment for better and more stable expert data generation. As shown in Figure 7 , without morphological variations, the gaits across different domains are similar, resulting in reduced data diversity. This aligns with our analysis on previous benchmarks. As shown in Figure 8 , 9 , 10 , it is clear that by introducing morphological variations during the data generation phase, the gaits and action modalities across different domains become substantially more diverse, thereby constructing a more challenging domain adaptation benchmark. Despite that, our proposed method is able to achieve state-of-the-art performance in environments with substantial dynamical gaps, demonstrating its broader applicability compared to prior approaches. Figure 7: Different Domain-specific Gaits in Hopper. Without morphological variations, the gaits are similar across different domains. Figure 8: Different Domain-specific Gaits in Walker Figure 9: Different Domain-specific Gaits in Ant Figure 10: Different Domain-specific Gaits in HalfCheetah D.3 Mastery Level In this section, we provide visualizations of the mastery. Here, mastery refers to a policy’s ability to successfully handle multiple domains, reflecting whether a single policy can robustly control different domains or embodiments, which directly impacts its practical effectiveness. As show in Figure 11 , in Walker2d, a policy with high mastery level is able to run forward rapidly and stably for an extended duration (top row). In contrast, policies that achieve non-zero returns but fail to reach mastery exhibit suboptimal gaits (middle row), or even collapse and fall (bottom row). A similar pattern can be observed in HalfCheetah, as illustrated in Figure 12 . Figure 11: Mastery Level Visualization in Walker2d Environments Figure 12: Mastery Level Visualization in HalfCheetah Environments Appendix E Pesudocodes In this section, we present the pesudocodes of the proposed DADP pipeline. Algorithm 1 Domain Adaptive Diffusion Policy (DADP) Domain Representation Learning Input : Offline datasets \mathcal{D} , context encoder E_{\phi} , forward dynamics model f_{\theta} , inverse dynamics model f_{\psi} for each epoch do Rebuild random domain-based paired index j=f(i) while epoch not finished \mathcal{D} do Sample paired data \tau^{i} , s^{j}_{t_{j}} , a^{j}_{t_{j}} , s^{j}_{t_{j}+1} Get context representation z^{i}=E_{\phi}(\tau^{i}) Get dynamical prediction \hat{s}^{j}_{t_{j}+1}=f_{\theta}(s^{j}_{t_{j}},a^{j}_{t_{j}},z^{i}) , \hat{a}^{j}_{t_{j}}=f_{\psi}(s^{j}_{t_{j}},s^{j}_{t_{j}+1},z^{i}) // Cross-Prediction Update \phi,\theta,\psi with loss: \mathcal{L}(\phi,\theta,\psi):=\beta_{\text{forward}}\left\|\hat{s}^{j}_{t_{j}+1}-s^{j}_{t_{j}+1}\right\|^{2}+\beta_{\text{inverse}}\left\|\hat{a}^{j}_{t_{j}}-a^{j}_{t_{j}}\right\|^{2} end while end for DADP Training Input: Dataset \mathcal{D} , Denoising Schedule \alpha_{1:K},\sigma_{1:K} , Guidance Scale \lambda , Denoising Policy \epsilon_{\xi} while not converged do 1. Data Sampling & Preparation Sample batch (\tau,z_{\text{pre}})\sim\mathcal{D} // Sample trajectories and precomputed representations Set x_{0}=\tau and M\ s.t.\ M_{t