Embodied Representation Alignment with Mirror Neurons

Wentao Zhu, Zhining Zhang, Yuwei Ren, Yin Huang, Yizhou Wang, [1.2ex] Center on Frontiers of Computing Studies, School of Compter Science, Peking University · 2025

Read Embodied Representation Alignment with Mirror Neurons by Wentao Zhu, Zhining Zhang, Yuwei Ren, Yin Huang, Yizhou Wang, [1.2ex] Center on Frontiers of Computing Studies, School of Compter Science, Peking University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21136.txt.

Introduction

Certain brain cells fire both when you wave and when you watch someone else wave; they are called mirror neurons. Wentao Zhu and five co-authors import that principle into machine learning in a 2025 paper for ICCV 2025. Their subject is embodied representation alignment for action understanding and execution.

Machine learning usually treats watching an action and performing it as separate tasks. The authors first show that intermediate representations for the two spontaneously align during learning. They then add an approach that aligns the representations explicitly. The unified view treats understanding and doing as two sides of one representation. Robots and video models alike could benefit, since an agent that represents its own actions like observed ones may learn skills from watching people more directly.

Catalog record

CREATORWentao Zhu, Zhining Zhang, Yuwei Ren, Yin Huang, Yizhou Wang, [1.2ex] Center on Frontiers of Computing Studies, School of Compter Science, Peking University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-21136

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Embodied Representation Alignment with Mirror Neurons Wentao Zhu, Zhining Zhang, Yuwei Ren, Yin Huang, Yizhou Wang, [1.2ex] Center on Frontiers of Computing Studies, School of Compter Science, Peking University Abstract Mirror neurons are a class of neurons that activate both when an individual observes an action and when they perform the same action. This mechanism reveals a fundamental interplay between action understanding and embodied execution, suggesting that these two abilities are inherently connected. Nonetheless, existing machine learning methods largely overlook this interplay, treating these abilities as separate tasks. In this study, we provide a unified perspective in modeling them through the lens of representation learning. We first observe that their intermediate representations spontaneously align. Inspired by mirror neurons, we further introduce an approach that explicitly aligns the representations of observed and executed actions. Specifically, we employ two linear layers to map the representations to a shared latent space, where contrastive learning enforces the alignment of corresponding representations, effectively maximizing their mutual information. Experiments demonstrate that this simple approach fosters mutual synergy between the two tasks, effectively improving representation quality and generalization. 1 Introduction “The body is our general medium for having a world.” — Maurice Merleau-Ponty Neuroscience research has uncovered a fascinating mechanism behind multiple cognitive abilities: mirror neurons . First identified in macaque monkeys, these neurons fire both when an individual observes an action and when they perform it themselves [ 4 , 9 , 32 ] . In essence, the neural representations of observed and executed actions are inherently aligned . Subsequent studies confirm similar systems in the human brain, where observing others’ actions activates corresponding motor regions, as if the observer were performing them [ 7 , 23 ] . This suggests that action understanding arises from neural simulations of observed behaviors, not merely abstract reasoning [ 33 ] . By mapping external movements onto its own motor repertoire, the brain internally “experiences” the action, enabling an intuitive grasp of others’ intentions [ 21 ] . Figure 1 : Conceptual demonstration of mirror neurons. These neurons activate both when observing an action and when performing it oneself, illustrating the shared neural representations that link perceptual and motor systems. This neural mechanism highlights the tight bond between two fundamental cognitive abilities: action understanding and embodied execution. Concretely, action understanding enables agents to interpret the meaning and intent behind others’ actions, while embodied execution allows them to physically interact with the environment to achieve goals. Crucially, these two abilities are deeply interconnected: action understanding supports embodied execution by guiding imitation learning and skill acquisition, while in turn, embodied execution provides firsthand sensorimotor experience that refines and deepens action understanding [ 6 , 26 ] . The biological foundation of this interplay is exemplified by mirror neurons, which also demonstrates the core idea of embodied cognition [ 3 , 8 , 1 , 40 , 36 ] —the notion that cognitive processes are not merely functions of the brain or abstract activities of the mind, but are deeply rooted in the body’s sensorimotor interactions with the world. Despite this biological synergy, current machine learning approaches typically address action understanding and embodied execution independently, overlooking their potential to inform and enhance each other [ 12 , 41 , 46 ] . This separation impedes the learning of generalizable and comprehensive action representations, which in turn limits performance on downstream tasks. In light of this, this work proposes to unify action understanding and embodied execution through the lens of representation learning. We begin by examining the relationship between the neural representations learned by models for these two tasks. Specifically, we investigate whether the representations align when observing and executing the same actions, how this alignment evolves during training, and its correlation with task success ( Section 2 ). Building on these insights, we propose a paradigm that jointly trains both models, bridging the gap between action understanding and embodied execution. The core idea is to project the agent’s representations of observed and executed actions into a shared latent space, where contrastive learning enforces their alignment ( Section 3 ). This approach explicitly aligns the representations of observed and executed actions, which is exactly what biological mirror neurons do. From the viewpoint of information theory, it is equivalent to maximizing the mutual information between the neural representations of the same semantic actions in action understanding and embodied execution. We evaluate our method on action recognition and multi-task object manipulation benchmarks, demonstrating that the proposed framework enables the two tasks to reinforce each other ( Section 4 ). Our results show that the learned representations are more disentangled and robust, leading to improved generalization. Furthermore, we investigate the impact of different alignment strategies, exploring the appropriate granularity for alignment. We summarize the key contributions of this paper below: • We conceptualize action understanding and embodied execution as a unified system, grounded in neuroscience and cognitive insights. • We discover that these two models exhibit representation alignment spontaneously, which correlates to some extent with task success. • We introduce a representation learning approach inspired by mirror neurons, which directly aligns the neural representations of observing and executing corresponding tasks, thereby effectively maximizing their mutual information. • Experiments demonstrate that our method is simple and effective, enhancing the generalization and representation learning quality for both tasks. 2 Probing Representation Alignment First, we aim to investigate whether action understanding and embodied execution models, when trained separately, exhibit neural representation alignment similar to mirror neurons. If such alignment exists, we further seek to understand the factors that influence it. 2.1 Problem Formulation Given an action understanding model \mathcal{U}(\cdot;\theta_{u}) and an embodied execution model \mathcal{E}(\cdot;\theta_{e}) , we aim to extract their intermediate neural representations and study their alignment. Specifically, the action understanding model \mathcal{U} takes a perceptual sequence V , such as a video, and produces an internal representation \bm{u} , from which it predicts the semantic label \hat{y}=\mathcal{U}(V) . Meanwhile, the embodied execution model \mathcal{E} takes the environment state S and an instruction I ( e.g . , a natural language command) as input, generating the representation \bm{e} , which is then decoded to the next action \hat{a}=\mathcal{E}(S,I) . The neural representations of observed and executed actions, \bm{u} and \bm{e} , originally reside in different high-dimensional representation spaces. In principle, they share some common information, such as the semantic concepts and spatial relationships of the scene. Meanwhile, they would also capture unique aspects specific to the task. For instance, \bm{u} may emphasize high-level action semantics and visual patterns essential for understanding, whereas \bm{e} is more attuned to physical constraints and feasibility, which are crucial for execution. Therefore, to effectively measure their alignment, we aim to learn a pair of linear transformations, \mathcal{T}_{u}(\cdot;\phi_{u}) and \mathcal{T}_{e}(\cdot;\phi_{e}) , that map \bm{u} and \bm{e} onto a shared latent space \mathbb{Z} and align them. Formally: \bm{z}_{u}=\mathcal{T}_{u}(\bm{u})\in\mathbb{Z},\hskip 10.00002pt\bm{z}_{e}=\mathcal{T}_{e}(\bm{e})\in\mathbb{Z}, (1) where \bm{z}_{u} and \bm{z}_{e} denote the aligned action understanding and embodied execution representations, respectively. Figure 2 : Alignment probing of off-the-shelf model representations. We extract internal neural representations from an Action Understanding (AU) model and an Embodied Execution (EE) model and train two linear transformations, \mathcal{T}_{u} and \mathcal{T}_{e} , to align them. Both models are pretrained separately and remain frozen during this process. 2.2 Probing

Read the complete text