Understanding Multimodal Failure in Action-Chunking Behavioral Cloning
Lorenzo Mazza [email protected], Massimiliano Datres [email protected], Ariel Rodriguez, Sebastian Bodenstedt, Gitta Kutyniok, Stefanie Speidel · 2026
Read Understanding Multimodal Failure in Action-Chunking Behavioral Cloning by Lorenzo Mazza [email protected], Massimiliano Datres [email protected], Ariel Rodriguez, Sebastian Bodenstedt, Gitta Kutyniok, Stefanie Speidel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22493.txt.
Introduction
Mazza and others ask why cloned robot policies fail when one view allows several right moves. For hidden variable policies, keeping each shown mode alive needs action linked detail in the hidden code. Heavy smoothing can erase that split. For direct action policies, smooth maps struggle to spread weight across far apart modes, so wide coverage needs sharp jumps or bridge zones. Fake steering tests and a real two mode robot job back the account.
Catalog record
| CREATOR | Lorenzo Mazza [email protected], Massimiliano Datres [email protected], Ariel Rodriguez, Sebastian Bodenstedt, Gitta Kutyniok, Stefanie Speidel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 120034 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22493
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Multimodal Failure in Action-Chunking Behavioral Cloning Lorenzo Mazza [email protected], Massimiliano Datres [email protected], Ariel Rodriguez, Sebastian Bodenstedt, Gitta Kutyniok, Stefanie Speidel Abstract Behavioral cloning becomes difficult when the same observation admits several valid actions. We study this problem for action-chunking policies and show that different multimodal parameterizations fail in different ways. For latent-variable policies, posterior–prior regularization makes deployment-time sampling more reliable, but excessive regularization removes the action-conditioned information needed to distinguish demonstrated modes. Reducing this regularization can preserve mode information, but then success depends on whether the prior covers the relevant latent regions. For action-space generative policies, multimodality is constrained by the smoothness of the base-to-action transport: a map with small Lipschitz constant cannot assign substantial probability to many well-separated modes. Covering many modes therefore requires either sharp transitions in base space or off-support bridge regions in action space. Experiments on synthetic multimodal tasks and robotic simulation benchmarks support these mechanisms. † † footnotetext: † Corresponding authors. 1 NCT/UCC Dresden, UKDD Dresden, TU Dresden, DKFZ Heidelberg. 2 Ludwig-Maximilians-Universität München, Munich Center for Machine Learning (MCML). 3 BMFTR Research Hub 6G-Life. 4 Cluster of Excellence CeTI. 5 University of Tromsø. 6 German Aerospace Center (DLR). 7 HZDR Dresden. 1 Introduction Imitation learning trains a robot policy directly from expert demonstrations [ 18 , 21 , 11 ] . A standard approach is behavioral cloning (BC), which fits a policy to observation–action pairs collected under an expert policy, without the need for online interaction, reward engineering, or exploration. Due to its practical appeal, BC has become a dominant paradigm for training deep visuomotor policies in robotic manipulation [ 31 , 6 , 28 , 15 , 2 ] . A central challenge in behavioral cloning is multimodality in the expert’s conditional action distribution: for the same observation, the dataset may contain multiple valid actions. This may arise from different expert styles, equally good choices, or latent factors absent from the observation space [ 17 ] . Under an L_{2} or unimodal-Gaussian behavioral-cloning head, the maximum-likelihood solution regresses to the conditional mean of the modes, which is typically not an element of any mode’s support [ 9 ] . A canonical example is inverse kinematics in robot control [ 1 ] : the same end-effector target may admit disconnected joint-space solutions. Averaging can yield a joint command that lies on none of the inverse-kinematics branches, while filtering to one branch removes feasible alternatives needed under constraints such as obstacles or joint limits. Contemporary multimodal BC methods typically predict short chunks of future actions rather than single-step actions, improving temporal coherence while enabling one-shot non-autoregressive inference over the chunk [ 31 , 6 ] . There are two broad approaches to making such policies multimodal: latent-variable methods and action-space generative methods , as illustrated in Fig. 4 . Latent-variable methods introduce an unobserved code, continuous or discrete, from which different action chunks can be decoded [ 31 , 16 ] . Action-space generative methods instead model the distribution over action chunks directly, using flows or diffusion models [ 6 , 2 ] . Both approaches can generate diverse actions, and their multimodality is typically evaluated by rollout performance. In both approaches, however, it remains unclear which model- and training-level quantities control the multimodality of the learned policy. Our goal is to fill this gap by providing a precise definition of multimodality. Furthermore, we identify the key quantities that must be controlled to preserve multimodality: the posterior–prior regularization strength in CVAE-based policies, and the Lipschitz constant of the base-to-action map in action-space generators. Contribution. We study how posterior-prior regularization affects multimodality in action-chunking behavioral cloning. For latent-variable policies, we show that preserving demonstrated modes requires action-conditioned latent information, see Proposition 1 . We then show that pointwise posterior–prior regularization can suppress this information as its strength increases, giving the multimodality collapse criterion in Corollary 2 . For action-space generators, we show that a deterministic sampler with small Lipschitz constant is more likely to assign a small probability to many modes, see Proposition 2 , losing multimodality. Therefore, regularization methods that enforce a small Lipschitz constant tend to suppress multimodality. We empirically validate our claims, and release the code used in the experiments. Structure of the paper. Section 2 defines multimodal behavioral cloning over actions and the generative policy families considered in the paper. Section 4 formalizes mode preservation and proves that latent policies must carry action-conditioned mode information, leading to a collapse criterion for pointwise prior regularization, a latent-geometry condition for aggregate matching, and bridge bounds for action-space generators. Section 5 introduces controlled synthetic benchmarks with ground-truth mode labels and evaluates representative methods on synthetic and robotic simulation tasks to test whether these mechanisms explain practical failures. 2 Background For any notation that may be unclear, we refer the reader to Appendix A . Offline imitation learning. We consider a finite-horizon Markov decision process (\mathcal{S},\mathcal{A},P,r,\rho_{0},\mathsf{T}) , where \mathcal{S} is the state or observation space, (\mathcal{A},d_{\mathcal{A}}) is the (metric) action space, P(\cdot\mid s,a)\in\mathcal{P}(\mathcal{S}) is the transition kernel, r:\mathcal{S}\times\mathcal{A}\to\mathbb{R} is the reward, \rho_{0}\in\mathcal{P}(\mathcal{S}) is the initial-state distribution, and \mathsf{T}\in\mathbb{N} is the horizon. A stochastic policy \pi:\mathcal{S}\to\mathcal{P}(\mathcal{A}) maps each state s to a probability distribution \pi(\cdot\mid s) over actions. In the offline-imitation-learning setting, the reward signal r is not observed. Instead, we are given demonstrations generated by an expert policy \pi_{E} . A trajectory realization is \tau=(s_{0},a_{0},\ldots,s_{\mathsf{T}-1},a_{\mathsf{T}-1},s_{\mathsf{T}})\in\mathcal{T}_{\mathsf{T}}:=(\mathcal{S}\times\mathcal{A})^{\mathsf{T}}\times\mathcal{S} . The expert policy, together with P and \rho_{0} , induces a trajectory law p_{E}\in\mathcal{P}(\mathcal{T}_{\mathsf{T}}) . When the relevant densities or probability mass functions exist, this law factorizes as p_{E}(\tau)=\rho_{0}(s_{0})\prod_{t=0}^{\mathsf{T}-1}\pi_{E}(a_{t}\mid s_{t})\,P(s_{t+1}\mid s_{t},a_{t}) . We denote by p_{\mathcal{D}}\in\mathcal{P}(\mathcal{S}\times\mathcal{A}) the time-marginal expert state-action distribution: p_{\mathcal{D}}(s,a):=\frac{1}{\mathsf{T}}\sum_{t=0}^{\mathsf{T}-1}p_{E}(S_{t}=s,A_{t}=a), where p_{E}(S_{t}=s,A_{t}=a) denotes the t -step state-action marginal induced by the trajectory law p_{E} . The goal of offline imitation learning is to learn a policy \pi_{\theta}(\cdot\mid s) from samples drawn from this expert state-action law. Let T_{1},\ldots,T_{N}\overset{\mathrm{iid}}{\sim}p_{E} be independent expert trajectory random variables. The dataset of expert demonstrations is one realization \tilde{\mathcal{D}}=\{\tau_{i}\}_{i=1}^{N}=\big\{(s^{i}_{0},a^{i}_{0},\ldots,s^{i}_{\mathsf{T}-1},a^{i}_{\mathsf{T}-1},s^{i}_{\mathsf{T}})\big\}_{i=1}^{N} . Using \tilde{\mathcal{D}} , we construct the empirical state-action distribution p_{\tilde{\mathcal{D}}}:=\frac{1}{N\mathsf{T}}\sum_{i=1}^{N}\sum_{t=0}^{\mathsf{T}-1}\delta_{(s_{t}^{i},a_{t}^{i})} . Remark 1 . In practice, action-chunk BC heads a_{t:t+\mathsf{H}-1}\in\mathcal{A}^{\mathsf{H}} are commonly used. Action-chunk BC is executed open-loop between observations, to retain temporal dependence absent from p_{\mathcal{D}} and reduce rollout error [ 31 , 26 ] . All definitions and results extend by replacing \mathcal{A} with \mathcal{A}^{\mathsf{H}} , so samples become (s_{t}^{i},a_{t:t+\mathsf{H}-1}^{i}) . Multimodal imitation learning. We formalize multimodality as a property of p_{\mathcal{D}}(\cdot\mid s) . For each observation s\in\mathcal{S} , the support of the expert conditional action distribution is contained in finitely many well-separated valid mode sets. In other words, for each s\in\mathcal{S} , there exists a natural number \mathsf{K}(s)\geq 1 and a family of covering sets C_{1}(s),\ldots,C_{\mathsf{K}(s)}(s)\subseteq\mathcal{A} of \operatorname{supp}p_{\mathcal{D}}(\cdot\mid s) such that \operatorname{supp}p_{\mathcal{D}}(\cdot\mid s)\subseteq\bigcup_{k=1}^{\mathsf{K}(s)}C_{k}(s),\qquad\Delta(s):=\min_{k\neq k^{\prime}}\operatorname{dist}_{\mathcal{A}}\!\bigl(C_{k}(s),C_{k^{\prime}}(s)\bigr)>0. (1) Each set C_{k}(s) represents one valid strategy or trajectory mode for observation s . The mode sets have disjoint closures. We further define the mode-assignment function g:\mathcal{A}\times\mathcal{S}\to\mathbb{N}_{0} as g(a,s)=\sum_{k=1}^{\mathsf{K}(s)}k\,\mathbf{1}_{\{a\in C_{k}(s)\}},\qquad\mathbf{1}_{\{a\in C_{k}(s)\}}=\begin{cases}1,&a\in C_{k}(s),\\ 0,&\text{otherwise}.\end{cases} (2) The function g is a mode-level labeling of expert actions used for the analysis. For each realized pair (s,a) , g(a,s) is the deterministic index of the mode containing a . Under the data distribution (S,A)\sim p_{\mathcal{D}} , we denote the induced mode-label random variable by M:=g(A,S). Here \mathsf{K}(s)=1 gives the unimodal