Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation

Xuening Zhang, Qi Lv, Xiang Deng, Miao Zhang, Xingbo Liu, Liqiang Nie · 2026

Read Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation by Xuening Zhang, Qi Lv, Xiang Deng, Miao Zhang, Xingbo Liu, Liqiang Nie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21051.txt.

Introduction

Zhang and co-authors present Cortical Policy, a dual-stream view transformer for robotic manipulation. View transformers read multi-view images to pick actions, but often reason weakly about 3D space. One stream aligns features of stable keypoints from a pretrained 3D model. The other stream models gaze, inspired by the brain's dorsal pathway, and adapts as scenes change. The two streams merge to choose final actions under language commands. On RLBench, COLOSSEUM, and real robot tasks, the method beats leading baselines.

Catalog record

CREATORXuening Zhang, Qi Lv, Xiang Deng, Miao Zhang, Xingbo Liu, Liqiang Nie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70985
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21051

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation Xuening Zhang, Qi Lv, Xiang Deng, Miao Zhang, Xingbo Liu, Liqiang Nie Abstract View transformers process multi-view observations to predict actions and have shown impressive performance in robotic manipulation. Existing methods typically extract static visual representations in a view-specific manner, leading to inadequate 3D spatial reasoning ability and a lack of dynamic adaptation. Taking inspiration from how the human brain integrates static and dynamic views to address these challenges, we propose Cortical Policy , a novel dual-stream view transformer for robotic manipulation that jointly reasons from static-view and dynamic-view streams. The static-view stream enhances spatial understanding by aligning features of geometrically consistent keypoints extracted from a pretrained 3D foundation model. The dynamic-view stream achieves adaptive adjustment through position-aware pretraining of an egocentric gaze estimation model, computationally replicating the human cortical dorsal pathway. Subsequently, the complementary view representations of both streams are integrated to determine the final actions, enabling the model to handle spatially-complex and dynamically-changing tasks under language conditions. Empirical evaluations on RLBench, the challenging COLOSSEUM benchmark, and real-world tasks demonstrate that Cortical Policy outperforms state-of-the-art baselines substantially, validating the superiority of dual-stream design for visuomotor control. Our cortex-inspired framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control. 1 Introduction Enabling robots to handle the uncertainty of unstructured, non-stationary environments remains a fundamental challenge for robotic manipulation ( 31 ; 7 ; 28 ) . Critically, this challenge requires coherent scene perception through robust fusion of multi-modal inputs, including vision, language, and proprioception ( 34 ) . To achieve this, view transformers ( 15 ; 14 ) provide an efficient solution by leveraging multi-view images to predict actions, showing competitive performance while offering greater scalability than explicit 3D representation-based approaches. Leveraging a set of static camera views around the robot workspace, previous view transformers typically extract visual representations through naively fusing view-specific 2D information. Despite demonstrated competence in stationary environments ( 15 ; 14 ; 49 ; 39 ) , this paradigm fails to model cross-view relationships, hampering 3D spatial understanding beyond 2D images. More importantly, the static camera configurations lack dynamic-view perception essential for human-like manipulation with unpredictable object displacements ( 18 ) . These limitations manifest as two frequent failure modes in robotic manipulation: inadequate spatial reasoning and dynamic adaptation failure . As shown in Fig. 1 (top), when placing an object in between the others, the SOTA view transformer ( 14 ) exhibits significant error, failing to place within the right region. This lack of robustness to 3D scene structure is further supported by findings that view transformers are sensitive to environmental disturbances like texture, lighting, and table color variations ( 39 ) . Furthermore, when the target object is moved during approach, existing methods ( 15 ; 14 ; 39 ) fail to adjust trajectories as expected, persisting with the originally-planned trajectory until task failure (Fig. 1 , bottom). These empirical findings underscore that the scene perception provided by current view transformers remains incomplete, which hinders robust manipulation performance. Figure 1: Deficiencies of prior view transformers for robotic manipulation. (Top) A task requiring the robot to understand the spatial relationships between two bottles before deciding on the placing position. Previous method RVT-2 fails to merge different camera views correctly in 3D, causing wrong block placement. (Bottom) Dynamic adaptation failure during object displacement. To bridge this gap, we draw inspiration from how the human brain organizes view-based visual cues to guide behavior, and in particular, two cortical pathways for visual processing. The ventral pathway utilizes static views for scene understanding, while the dorsal pathway specializes in dynamic-view perception, leveraging real-time visual feedback to adjust trajectory ( 41 ; 4 ) . Translating this cortical principle into a computational framework, we propose Cortical Policy, a dual-stream view transformer for endowing robots with integrated 3D spatial understanding and dynamic adaptation. Our method enhances robotic perception through two separate, complementary pipelines: a static-view stream that encodes enduring environmental structures and a dynamic-view stream that derives actions from motion cues. To enhance spatial comprehension, the static-view stream learns 3D-aware features by enforcing cross-view geometric consistency, which is supervised by a 3D foundation model. To facilitate adaptive re-planning under task dynamics, the dynamic-view stream extracts action-oriented features and heatmaps from a pretrained, position-aware transformer adapted from an egocentric gaze estimation model. By integrating view representations of both streams, Cortical Policy generates actions that are simultaneously geometrically grounded and dynamically adaptive. Extensive evaluations on RLBench, COLOSSEUM, and real-world tasks demonstrate that our cortex-inspired policy, with enhanced 3D awareness and adaptive motion control, substantially outperforms state-of-the-art baselines. Specifically, it exhibits superior robustness against environmental perturbations and boosts the interactive capabilities of an embodied agent in dynamic physical environments. The main contributions of this work are summarized as follows: • Different from prior view transformers that perform single-stream processing on static views, we propose Cortical Policy, a dual-stream view transformer that integrates static and dynamic views for robotic manipulation, mirroring the two human cortical pathways to advance visuomotor imitation learning. • Unlike view-independent processing in prior methods, we introduce a cross-view geometric consistency learning objective. This objective leverages a pretrained 3D foundation model (VGGT) to align cross-view features in a shared 3D space, significantly enhancing the spatial reasoning robustness of the static-view stream. • A novel dynamic-view stream absent in prior work is designed to emulate the human dorsal pathway. This stream extracts action-oriented representations from a position-aware, pretrained gaze estimation model, thereby enabling adaptive trajectory adjustment. 2 Related work This work extends view transformers for robotic manipulation by enhancing static-view 3D perception and introducing dynamic-view processing. We review the relevant work in this section. View Transformers for Robotic Manipulation. View transformers have become a prevalent architecture for language-conditioned manipulation ( 17 ; 35 ) . They aggregate multi-view visual inputs with language instructions and proprioception to predict 6-DoF gripper poses, states, and collision indicators. RVT ( 15 ) establishes a five-camera paradigm (back, front, top, left, right) to render virtual static views, using a view transformer to predict view-specific heatmaps, which are back-projected to 3D to estimate gripper translation; multi-camera features are concatenated to predict the remaining action components. To improve precision, VIHE ( 46 ) and RVT-2 ( 14 ) adopt multi-stage refinement: VIHE iteratively renders virtual in-hand static views, while RVT-2 localizes regions of interest with three static views (front, top, right) before predicting poses from refined regions. Recent methods enhance static-view visual representations with visual foundation models ( 49 ; 12 ) or 3D multi-view pretraining ( 39 ) . Although these methods have advanced static-view perception, their inherent reliance on pre-defined viewpoints limits the adaptability in dynamic scenarios. In contrast, Cortical Policy jointly leverages static and dynamic views for action prediction to overcome this limitation. 3D Perception in Robotics. To enhance robots’ understanding of the physical world, extensive efforts have been made to integrate 3D representations into robotic manipulation ( 22 ; 15 ; 33 ) . Existing approaches, however, face distinct challenges. Voxel-based methods ( 22 ; 43 ) are computationally expensive. Point cloud methods handle occlusion and sim-to-real transfer well, yet require fine-grained semantic alignment ( 50 ; 8 )

Read the complete text