Towards Fusing Point Cloud and Visual Representations for Imitation Learning
Atalay Donat, Xiaogang Jia, Xi Huang, Aleksandar Taranovic, Denis Blessing, Ge Li, Hongyi Zhou, Hanyi Zhang, Rudolf Lioutikov, Gerhard Neumann · 2025
Read Towards Fusing Point Cloud and Visual Representations for Imitation Learning by Atalay Donat, Xiaogang Jia, Xi Huang, Aleksandar Taranovic, Denis Blessing, Ge Li, Hongyi Zhou, Hanyi Zhang, Rudolf Lioutikov, Gerhard Neumann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12320.txt.
Introduction
Atalay Donat and co-authors wrote a 2025 paper on robot learning that joins two kinds of sight. One kind gives a 3D cloud of points, which captures shape well. The other gives flat color images, which carry texture and meaning that point clouds miss.
Past fusions pasted image traits onto points and lost the wide scene context. The new method, FPV-Net, feeds both wide and local image tokens into the point cloud coder by way of adaptive scaling. Tests on the RoboCasa bench, a hard set of home tasks, showed the limits of each sense alone. The fused method set the best scores across all tasks in the bench.
Catalog record
| CREATOR | Atalay Donat, Xiaogang Jia, Xi Huang, Aleksandar Taranovic, Denis Blessing, Ge Li, Hongyi Zhou, Hanyi Zhang, Rudolf Lioutikov, Gerhard Neumann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12320
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Fusing Point Cloud and Visual Representations for Imitation Learning Atalay Donat, Xiaogang Jia, Xi Huang, Aleksandar Taranovic, Denis Blessing, Ge Li, Hongyi Zhou, Hanyi Zhang, Rudolf Lioutikov, Gerhard Neumann Abstract Learning for manipulation requires using policies that have access to rich sensory information such as point clouds or RGB images. Point clouds efficiently capture geometric structures, making them essential for manipulation tasks in imitation learning. In contrast, RGB images provide rich texture and semantic information that can be crucial for certain tasks. Existing approaches for fusing both modalities assign 2D image features to point clouds. However, such approaches often lose global contextual information from the original images. In this work, we propose FPV-Net, a novel imitation learning method that effectively combines the strengths of both point cloud and RGB modalities. Our method conditions the point-cloud encoder on global and local image tokens using adaptive layer norm conditioning, leveraging the beneficial properties of both modalities. Through extensive experiments on the challenging RoboCasa benchmark, we demonstrate the limitations of relying on either modality alone and show that our method achieves state-of-the-art performance across all tasks. † † affiliationnotice: Equal contribution 1 Introduction Imitation Learning (IL) has become a fundamental approach in robotic learning ( Brohan et al., 2022 ; Chi et al., 2023 ; Zhao et al., 2023 ; Black et al., 2024 ; Kim et al., 2024 ) , allowing agents to acquire complex behaviors by mimicking expert demonstrations. IL can additionally benefit from contextual information that provides task description, therefore reducing the need for inferring task goal from the demonstrations ( Ding et al., 2019 ) . A crucial aspect of IL is the choice of the used input representation, as it directly impacts the agent’s ability to generalize and make informed decisions. RGB images are a common input modality because they offer rich texture and semantic information that can be critical for tasks involving object recognition and contextual reasoning ( Mandlekar et al., 2021 ; Reuss et al., 2024b ; Liu et al., 2024 ) . Additionally, they are easy to obtain and relatively cheap, making them a practical choice in many scenarios. Another input modality is a point cloud ( Zhu et al., 2024 ; Ze et al., 2024 ; Ke et al., 2024 ) , which provides us with geometric information. Point cloud representations have proven highly effective for robotic manipulation due to their ability to directly encode 3D spatial structures. A further modality are language instructions. They contain relevant task context ( Stepputtis et al., 2020 ; Li et al., 2023 ; Reuss et al., 2024b ) , such as human understandable task descriptions. All these input types provide different benefits and limitations in the learning process, and we should fuse them appropriately to extract all the individual benefits, while offsetting the limitations. Therefore, fusing different modalities is a relevant but challenging problem. In this paper, we focus on the fusion of RGB images and point clouds while also taking language instructions into account. Despite their complementary nature, integrating these RGB images and point clouds remains a significant challenge in IL. Existing approaches ( Gervet et al., 2023 ; Shridhar et al., 2023 ; Ze et al., 2024 ) primarily attempt to assign 2D visual features to point clouds, thereby incorporating RGB information into 3D representations. However, such strategies often fail to retain the global contextual information from images, leading to suboptimal performance in tasks that require both precise spatial reasoning and high-level semantic understanding. As a result, neither modality alone—nor naïve fusion techniques—achieves universally strong performance across diverse imitation learning benchmarks. Yet, more recent approaches of combining modalities such as adaptive conditioning in Layer-Norm layers ( Peebles & Xie, 2022 ) has not yet been explored in the imitation learning context, even though it allows a more flexible sensor fusion scheme. Figure 1 : Processing each input modality to generate corresponding embeddings. Top : A FiLM-ResNet architecture is used to extract a feature map from the context image. The feature map is processed through average pooling and flattening to obtain global and local feature tokens, which are then concatenated and fed into the transformer along with a learnable CLS token, whose output is used as a condition vector for the diffusion policy (Figure 2 ). Middle : The point cloud input is processed by applying FPS to sample points, followed by KNN to group point patches using these FPS points as centers. The resulting patches are passed through a point patches encoder, which can be a lightweight MLP or the pretrained SUGAR model. Bottom : The CLIP model is employed to generate the language embedding for the behavior prompt. Figure 2 : Conditioned on image CLS tokens, the transformer-based diffusion policy (DiT block) denoises action chunk tokens by utilizing 3D point cloud tokens and language tokens as inputs. The conditioning process is detailed within the structure of the DiT block. To address this limitation, we introduce F usion of P oint Cloud and V isual Representation Net work (FPV-Net), a novel imitation learning method designed to effectively align and balance the strengths of both point cloud and RGB images. Our approach leverages novel conditioning methods for sensor fusion ( Peebles & Xie, 2022 ) and ensures that the geometric precision of point clouds is preserved while leveraging the global semantic richness of RGB inputs, enabling a more robust and generalizable policy learning process. For the extraction of representations from RGB images, we use a neural network based on the FiLM-ResNet architecture ( Perez et al., 2018 ) . This extraction process is conditioned on the language instruction, thus effectively incorporating this modality into our method. Moreover, we make use of both local features and global features, which we show to be critical for the manipulation tasks. To extract data from point clouds, we apply Furthest Point Sampling ( Eldar et al., 1994 ) and k-Nearest Neighbors, that are then encoded into learned embeddings. For fusing the modalities, we explore 3 different approaches, and show that fusing Point Cloud and Language as main modalities while using RGB images as the conditional modality using AdaLN conditioning ( Peebles & Xie, 2022 ) performs best. Figure 1 illustrates how FPV-Net extracts features from different modalities. We evaluate FPV-Net on RoboCasa ( Nasiriany et al., 2024 ) , a challenging benchmark for robotic manipulation. We conduct extensive experiments to analyze the impact of different input modalities. Our results indicate that neither point clouds nor RGB images alone provide optimal performance across all tasks, whereas naïve fusion methods often degrade performance due to poor alignment between modalities. FPV-Net consistently outperforms state-of-the-art approaches ( Ke et al., 2024 ; Ze et al., 2024 ) across all tasks, establishing a new benchmark in multimodal imitation learning. To summarize, our main contributions are threefold. First, we conduct systematic experiments on RoboCasa, showing that neither RGB images nor point clouds alone are sufficient for strong performance, as each modality excels in some tasks but performs poorly in others. Second, we introduce FPV-Net, a diffusion-based multi-modal imitation learning method that leverages point cloud inputs as the main modality and visual inputs as a conditional modality, integrated via AdaLN conditioning ( Peebles & Xie, 2022