Diverse Yet Consistent: Context-Guided Diffusion with Energy-Based Joint Refinement for Multi-Agent Motion Prediction Lei Chu, Yuhuan Zhao Abstract Deep generative models have become a promising approach for human motion prediction due to their ability to capture multimodal distributions and represent diverse human behaviors. However, generating predictions that are both diverse and jointly consistent among interacting agents remains challenging. In addition, most existing approaches are primarily evaluated using single-agent (marginal) metrics, which fail to fully reflect the joint dynamics of multi-agent interactions. We propose a diffusion-based framework that improves multi-agent motion prediction by leveraging rich contextual information from historical trajectories. This information is incorporated through a guidance mechanism to enhance the diversity and expressiveness of predicted motions. To further enforce interaction consistency, we introduce an energy-based formulation that refines the joint trajectory distribution while preserving the plausibility of individual trajectories. Extensive experiments on four benchmark datasets demonstrate that our approach consistently outperforms existing methods. Notably, our approach substantially improves both marginal (ADE/FDE) and joint (JADE/JFDE) metrics on ETH/UCY over strong marginal baselines. Compared with prior joint prediction methods, it delivers significant gains in marginal metrics while maintaining competitive joint performance. 1 Introduction Human trajectory forecasting aims to predict future human movements while accounting for the uncertainty and diversity of possible behaviors 2 ; 9 ; 51 ; 46 ; 31 ; 34 . It plays a critical role in applications such as autonomous driving 49 ; 46 ; 8 ; 30 ; 7 , digital health 25 ; 31 , and human–robot interaction 47 ; 56 ; 66 ; 57 , where accurately anticipating pedestrian motion is essential for safe decision-making. In multi-agent environments 46 ; 23 , this task requires predicting a distribution of possible future trajectories by modeling both individual motion histories and interactions among agents 67 . Despite significant progress, the problem remains challenging due to the inherent multi modality of human motion, where the same observed past can correspond to multiple plausible future paths across different environments 22 . Classical generative models, such as autoencoder-based approaches, address this challenge by learning compact latent representations of past trajectories and using them to decode and predict future motion. These models efficiently capture motion patterns and can be trained in unsupervised or self-supervised settings, making them easy to extend with techniques such as variational autoencoders 52 ; 61 , attention mechanisms 19 ; 55 , or social interaction modules 2 ; 35 ; 61 . However, they often have difficulty capturing multimodal future possibilities, which can lead to averaged trajectories and accumulated errors in long-term predictions. In addition, modeling complex multi-agent interactions often requires extra architectural components. In contrast, diffusion-based generative models 5 ; 50 ; 26 ; 33 ; 24 ; 13 naturally model multimodal trajectory distributions by generating multiple plausible futures instead of a single averaged prediction. They can better capture complex motion patterns and interactions, producing diverse and realistic trajectories in crowded or uncertain environments. Nevertheless, diffusion models can exhibit temporal inconsistencies, where stochastic denoising introduces small jitters between time steps, and goal ambiguity, generating trajectories that appear locally plausible but are globally inconsistent with the underlying intent. To overcome the challenges discussed above, we propose a deep generative modeling approach for human trajectory prediction. The main contributions of this paper are summarized as follows: • We propose CODA , a novel framework for consistent and diverse multi-agent motion prediction. Our approach enhances trajectory generation by incorporating rich contextual information from historical observations and integrating it into the generative process to improve prediction diversity and expressiveness. • We introduce a simple yet effective energy-based formulation 11 for joint trajectory refinement, which preserves the plausibility of individual trajectories while improving joint consistency among interacting agents. • Extensive experiments on four human motion datasets demonstrate that CODA achieves state-of-the-art performance across multiple metrics, effectively balancing the trade-off between marginal metrics (ADE/FDE) and joint metrics (JADE/JFDE) 54 , highlighting the importance of modeling diverse yet consistent multi-agent motion. (a) (b) (c) (d) Figure 1 : Core concept and result of CODA: By incorporating rich interaction context and applying energy-based optimization, CODA improves joint behavior while preserving marginal accuracy, achieving the best performance on marginal metrics (ADE/FDE) and mean Collision Rate (CR), and the second-best results on joint metrics (JADE/JFDE). 2 Related Works 2.1 Human Trajectory Prediction Human trajectory prediction aims to forecast future pedestrian positions from observed motion histories and surrounding interactions. Early approaches relied on physics-based and probabilistic models, such as the Social Force Model 42 , Kalman Filters 34 , and Hidden Markov Models 38 ; 12 , which describe motion using predefined dynamics but struggle to capture complex behaviors. With the advancement of deep learning, RNN- and LSTM-based models were introduced to learn temporal dependencies in trajectory sequences. To better model social interactions, subsequent work proposed interaction-aware architectures such as Social-LSTM 2 and graph-based methods using Graph Neural Networks (GNNs) 35 . More recently, attention mechanisms and transformer-based models 14 ; 48 have been explored to capture richer spatial–temporal dependencies and long-range interactions. In parallel, generative approaches, including GANs, VAEs, and diffusion models, have been developed to produce multiple plausible future trajectories, addressing the multi-modality of human motion prediction. 2.2 Interaction Modeling and Enhancement Existing methods for interaction modeling in human trajectory prediction can be broadly categorized into several groups based on how agent interactions are represented. Rule-based approaches rely on hand-crafted formulations, such as the Social Force Model 42 , which describe pedestrian interactions using predefined physical rules 18 . Neighborhood-based aggregation methods capture local interactions by pooling information from nearby agents, as exemplified by social pooling 2 . Pairwise interaction learning methods explicitly model relationships between agent pairs based on their relative positions and motion patterns 63 ; 68 ; 21 . More recent work adopts graph-based methods, where Graph Neural Networks (GNNs) represent pedestrians as nodes and their interactions as edges 53 , enabling flexible modeling of dynamic crowd structures. Finally, attention-based and transformer-based approaches further enhance interaction modeling by selectively focusing on relevant agents and capturing long-range spatial–temporal dependencies 19 ; 35 ; 37 . Building upon these developments, our work introduces richer contextual representations for diffusion-based trajectory generation, enabling diverse yet consistent multi-modal predictions. 2.3 Generation with Guidance Guided generation in diffusion models is commonly implemented through classifier guidance, which adds gradients from an external classifier trained on noisy samples, and classifier-free guidance (CFG) 17 , which combines conditional and unconditional score estimates without requiring a separate classifier. More generally, model- or regressor-guidance methods bias 16 ; 17 ; 20 sampling using gradients from learned constraint models. In trajectory prediction, guidance is often realized through goal/intent guidance and scene/map guidance 45 ; 28 ; 1 , which encourage trajectories to follow likely destinations while respecting environmental constraints 40 ; 64 . In this work, as illustrated in Fig. 1 , we adopt classifier-free guidance and incorporate richer contextual information during generation, while further refining the joint distribution of sampled trajectories with an energy-based model 11 ; 39 ; 36 , preserving individual trajectory plausibility while improving multi-agent consistency 54 . 3 Approach 3.1 Trajectory Prediction Problem Multi-agent trajectory forecasting aims to predict the future motion of multiple interacting agents based on their historical observations and scene context. Consider a dynamic scene containing N agents observed over T_{h} time steps. The historical trajectories of all agents are represented as \mathbf{X}_{1:T_{h}}=\{\mathbf{x}_{i}^{t}\mid i=1,\dots,N,\;t=1,\dots,T_{h}\}, (1) where \mathbf{x}_{i}^{t}\in\mathbb{R}^{2} denotes the 2D spatial position of agent i at time step t . Given these observations, the objective is to predict the future trajectories of all agents over the next T_{f} time steps, \mathbf{Y}_{T_{h}+1:T_{h}+T_{f}}=\{\mathbf{y}_{i}^{t}\mid t=T_{h}+1,\dots,T_{h}+T_{f}\}, (2) where \mathbf{y}_{i}^{t}\in\mathbb{R}^{2} denotes the ground-truth future position of agent i at time step t . The trajectory forecasting task can therefore be formulated as learning a model f_{\theta}(\cdot) that estimates the conditional distribution P_{\theta}(\mathbf{Y}_{T_{h}+1:T_{h}+T_{f}}\mid\mathbf{X}_{1:T_{h}},\mathcal{C}), (3) where \mathcal{C} represents additional contextual information such as scene semantics, map priors, and agent interactions. Since future trajectories are inherently uncertain and often multi-modal, the model typically predicts a set of K possible future trajectories \hat{\mathbf{Y}}^{(k)}_{T_{h}+1:T_{h}+T_{f}},\quad k=1,\dots,K, each corresponding to a plausible future motion hypothesis. In this section, we introduce the proposed method ( Fig. 2 ), CODA, a diffusion-based framework for joint trajectory modeling. CODA consists of three key modules: (1) Dynamic Context as Guidance Condition (DCGC), which extracts agents’ dynamic features as guidance conditions; (2) the Adaptive Condition Integration Module (ACIM), which injects the dynamic context as additional noise to guide embedding generation during diffusion; and (3) Joint Distribution Refinement (JDR), which shifts probability mass toward jointly consistent trajectories while suppressing trajectories that are individually plausible but jointly inconsistent. 3.2 Dynamic Context as Guidance Condition Recent studies construct guidance conditions from either agent interaction features (non-stationary) 19 ; 53 or dynamic features (stationary) 18 ; 1 ; 12 to guide trajectory generation. However, both approaches have inherent limitations. Non-stationary conditions, represented by dynamic feature embeddings, improve trajectory diversity but often introduce redundant information that may cause deviations from the agent’s true intentions. In contrast, stationary conditions maintain intention consistency through agent interest embeddings, but typically rely on a fixed number of representations, which may fail to capture varying intention structures across agents and thus limit trajectory diversity. In this work, inspired by 65 ; 62 , agent-wise features are obtained by applying self-attention 29 to embedded historical trajectories. Additionally, dynamic interaction features are extracted using the context Transformer 62 . Figure 2 : Illustrates the framework of our CODA. It consists of three key modules: (1) Dynamic Context as Guidance Condition (DCGC), which captures agents’ dynamic features as guidance conditions; (2) the Adaptive Condition Integration Module (ACIM), which incorporates agent dynamic context from these guidance conditions as additional noise to generate the next embedding during the diffusion generation phase; and (3) Joint Distribution Refinement (JDR), which shifts probability mass toward jointly consistent trajectories while reducing probability assigned to trajectories that are individually plausible but jointly inconsistent. 3.2.1 Agent-wise Context Extraction Given the embedded representation of the historical inputs \mathbf{X} , we employ a self-attention mechanism 29 to capture dependencies within the sequence. Specifically, a two-layer multilayer perceptron (MLP) is used to compute an attention score matrix that measures the relative importance of different historical observations. The weighted score matrix \mathcal{W}\in\mathbb{R}^{A\times K} is computed as \mathcal{W}=\mathrm{Softmax}\left(\mathrm{MLP}_{4d\times K}\left[\tanh\left(\mathrm{MLP}_{d\times 4d}\left(\mathbf{X}+\mathcal{P}\right)\right)\right]\right), (4) where \mathcal{P} denotes the positional embedding and k represents the number of extracted features, corresponding to the number of generated trajectory hypotheses. The historical information is then aggregated using the weighted score matrix to obtain the interest feature representation: \mathbf{G}_{1}\in\mathbb{R}^{k\times N\times d}=\mathcal{W}^{\top}\otimes\mathbf{X}. (5) 3.2.2 Global Context Extraction The global context (dynamic interaction) embedding among agents is extracted using four standard Transformer blocks, similar to the architecture used in 62 . Each block consists of a multi-head self-attention layer followed by a feed-forward network. The resulting global context representation is given by \mathbf{G}_{2}\in\mathbb{R}^{1\times N\times d}=\mathrm{Transformer}(\mathbf{X}), (6) where \mathbf{G}^{2} denotes the learned dynamic interaction embedding and \mathrm{Transformer}(\cdot) represents the stacked Transformer blocks that model dependencies among agents. We repeat \mathbf{G}^{2} for K times for subsequent use. Equations Eq. 5 and Eq. 6 describe the extraction of the agent’s inherent motion features and mutual interaction features, corresponding to approximations of long-term motion patterns and short-term interaction dynamics. Their combination captures the agent’s true motion intention and guides the model to generate realistic trajectory embeddings in the noise space. To preserve the individuality of both features while enriching the guidance signal, we concatenate them to form the final guidance condition: \mathbf{G}\in\mathbb{R}^{K\times A\times 2d}=Concat(\mathbf{G}_{1},\mathbf{G}_{2},dim=0), (7) 3.3 Adaptive Context Integration Existing diffusion-based methods typically employ either MLPs or Transformer encoders for noise (or target) prediction. MLP-based approaches can improve trajectory diversity but lack explicit interaction with guidance conditions, which may introduce bias. In contrast, Transformer-based methods integrate noise representations with agent interaction features through attention weighting, effectively fitting noise in the agent intention space rather than the target space, thereby limiting trajectory diversity. To address this limitation, we propose an Adaptive Condition Integration Module (ACIM) that dynamically injects the guidance condition \bf G into the noisy target embedding via cross-attention, thereby strengthening conditional guidance during the diffusion process: x_{t}^{G}=\mathrm{Softmax}\!\left(\frac{[(x_{t}+t_{E}+A_{E})W^{Q}(GW^{K})^{\top}]}{\sqrt{d}}\right)(GW^{V}), (8) where t_{E} and A_{E} denote the time-step embedding and agent-order embedding, respectively. The guidance-enhanced representation is then fused with the original noisy embedding as x_{t}=\mathrm{Concat}\left(x_{t},x_{t}^{G},\mathrm{dim}=-1\right). (9) 3.4 Optimization with Joint Distribution Refinement 3.4.1 Context Guidance based Prediction Given contextual information \mathbf{G} , the goal is to estimate the conditional marginal distribution of the target variable \mathbf{X} , denoted as p(\mathbf{X}\mid\mathbf{G}) . We adopt a conditional diffusion model 17 ; 40 that learns this distribution through a gradual denoising process. Specifically, a forward diffusion process progressively perturbs the data by adding Gaussian noise: q(\mathbf{x}_{t}\mid\mathbf{x}_{t-1})=\mathcal{N}\!\left(\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}\right),\quad t=1,\dots,T, (10) which transforms the data distribution into an isotropic Gaussian. The reverse process learns to recover the clean sample conditioned on context \mathbf{G} : p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{G}), (11) parameterized by a neural network that predicts the noise component \epsilon_{\theta}(\mathbf{x}_{t},t,\mathbf{G}) . The model is trained by minimizing the denoising objective \mathcal{L}=\mathbb{E}_{t,\mathbf{x}_{0},\epsilon}\left[\|\epsilon-\epsilon_{\theta}(\mathbf{x}_{t},t,\mathbf{G})\|^{2}\right], (12) where \epsilon\sim\mathcal{N}(0,\mathbf{I}) . Through iterative denoising conditioned on \mathbf{G} , the model learns to sample from the target distribution p(\mathbf{X}\mid\mathbf{G}) . To follow the trajectory prediction formulation commonly used in the literature 28 ; 5 ; 13 ; 22 , we convert the noise prediction into target prediction and incorporate a best-of-K estimation along with a diversity-encouraging loss: \displaystyle L_{\mathrm{reg}}= \displaystyle\left\|\mathbf{Y}-\hat{\mathbf{Y}}(t,\mathbf{G})\right\|_{2}^{2}+\min_{k\in\{1,\ldots,K\}}\left\|\mathbf{Y}-\widehat{\mathbf{Y}}^{(k)}\right\|_{2}^{2} (13) \displaystyle+\mathcal{L}_{\mathrm{div}}(\hat{\mathbf{Y}},\mathbf{Y}) In Eq. 13 , the first term can be interpreted as a variant of Eq. 12 28 ; 5 ; 13 ; 22 . The second term corresponds to the well-known best-of- K estimation, while the final term represents the diversity loss. To encourage diversity among the K predicted trajectory modes, we consider the temporal differences between consecutive predictions: \Delta\hat{y}_{i}=\hat{y}_{i+1}-\hat{y}_{i} and \Delta y_{i}=y_{i+1}-y_{i}. The diversity loss is defined as the Kullback-Leibler (KL) divergence between the normalized differences of predicted and ground-truth trajectories: \displaystyle\mathcal{L}_{\mathrm{div}}(\hat{\mathbf{Y}},\mathbf{Y}) \displaystyle=D_{\mathrm{KL}}\!\left(\sigma(\Delta\hat{\mathbf{y}})\;\|\;\sigma(\Delta\mathbf{y})\right) (14) \displaystyle=\sum_{i=1}^{T_{f}-1}\sigma(\Delta\hat{\mathbf{y}}_{i})\log\frac{\sigma(\Delta\hat{\mathbf{y}}_{i})}{\sigma(\Delta\mathbf{y}_{i})}. (15) The normalization function \sigma(\cdot) is defined using a temperature-scaled softmax: \displaystyle\sigma(\Delta y)_{i,j,k,l} \displaystyle=\frac{\exp(\Delta y_{i,j,k,l}/\tau)}{\sum_{j^{\prime}=1}^{K}\sum_{k^{\prime}=1}^{N}\sum_{l^{\prime}=1}^{T_{f}}\exp(\Delta y_{i,j^{\prime},k^{\prime},l^{\prime}}/\tau)}, (16) \displaystyle\sigma(\Delta\hat{y})_{i,j,k,l} \displaystyle=\frac{\exp(\Delta\hat{y}_{i,j,k,l}/\tau)}{\sum_{j^{\prime}=1}^{K}\sum_{k^{\prime}=1}^{N}\sum_{l^{\prime}=1}^{T_{f}}\exp(\Delta\hat{y}_{i,j^{\prime},k^{\prime},l^{\prime}}/\tau)}. (17) 3.4.2 Joint Distribution Refinement When the diffusion parameterization factorizes across agents, training yields accurate marginal distributions for each agent but does not explicitly capture their joint dependencies. This limitation motivates us to advocate joint distribution refinement (JDR). In this work, we build on the observation that diffusion models define a distribution over trajectories and can learn accurate marginals. To address the missing joint dependencies, we introduce an energy-based model (EBM) 11 , parameterized by \theta , that provides a joint correction to the diffusion distribution. The resulting composed distribution is P_{\text{joint}}(\mathbf{Y}\mid\mathbf{G})\propto\underbrace{P_{\text{Diff}}(\mathbf{Y}\mid\mathbf{G})}_{\text{good marginals}}\underbrace{\exp\!\left(-E_{\theta}(\mathbf{Y},\mathbf{G})\right)}_{\text{joint refinement}}, (18) where P_{\text{Diff}}(\mathbf{Y}\mid\mathbf{G}) provides accurate marginal distributions via the diffusion model, whereas the energy term \exp(-E_{\theta}(\mathbf{Y},\mathbf{G})) acts as an EBM-based correction to better capture inter-agent dependencies . Taking the logarithm of Eq. 18 and focusing on the joint refinement part, we obtain {{\cal L}_{{\rm{EBM}}}}=\mathop{\min}\limits_{\theta}\sum\limits_{i}{{E_{\theta}}}({{\bf{Y}}_{i}},{{\bf{G}}_{i}})+\sum\limits_{i