Cambrian- P : Pose-Grounded Video Understanding
Jihan Yang 1 ∗ Zifan Zhao 1 ∗ Xichen Pan 1 Shusheng Yang 1 Junyi Zhang 2 Bingyi Kang 1 Hu Xu 3 Saining Xie 1 1 New York University 2 UC Berkeley 3 Meta FAIR · 2026
Read Cambrian- P : Pose-Grounded Video Understanding by Jihan Yang 1 ∗ Zifan Zhao 1 ∗ Xichen Pan 1 Shusheng Yang 1 Junyi Zhang 2 Bingyi Kang 1 Hu Xu 3 Saining Xie 1 1 New York University 2 UC Berkeley 3 Meta FAIR (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22819.txt.
Introduction
Yang and eight colleagues build Cambrian-P, a video model that tracks camera position and direction across frames instead of treating each frame alone. Camera pose gives every view a shared spatial frame, yet most video models ignore it. Their model adds learnable camera tokens and a pose prediction head, gaining several percent on spatial reasoning tests and reaching top streaming pose scores on ScanNet.
Catalog record
| CREATOR | Jihan Yang 1 ∗ Zifan Zhao 1 ∗ Xichen Pan 1 Shusheng Yang 1 Junyi Zhang 2 Bingyi Kang 1 Hu Xu 3 Saining Xie 1 1 New York University 2 UC Berkeley 3 Meta FAIR |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70673 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22819
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cambrian- P : Pose-Grounded Video Understanding Jihan Yang 1 ∗ Zifan Zhao 1 ∗ Xichen Pan 1 Shusheng Yang 1 Junyi Zhang 2 Bingyi Kang 1 Hu Xu 3 Saining Xie 1 1 New York University 2 UC Berkeley 3 Meta FAIR Abstract Camera pose matters. The position and orientation of each viewpoint define a shared spatial coordinate frame that relates observations across video frames. Yet this signal is largely absent from multimodal LLMs (MLLMs) for video understanding, which process frames as isolated 2D snapshots, instead of the persistent scene humans perceive. We revisit pose as a lightweight supervisory signal and introduce Cambrian- P , a video MLLM augmented with per-frame learnable camera tokens and a pose regression head. With a carefully designed sampling scheme, the model achieves substantial gains of 4.5–6.5% on spatial reasoning benchmarks such as VSI-Bench, generalizes across eight additional spatial and general video QA benchmarks, and, as a byproduct, achieves state of the art streaming pose estimation on ScanNet. Surprisingly, training on pseudo-annotated poses from in-the-wild video further improves general video QA benchmarks, showing pose helps beyond spatial reasoning. Together, these results position camera pose as a fundamental signal for video models that reason about the physical world. 1 1 footnotetext: JY led the project, JY and ZZ contributed equally. Website https://cambrian-mllm.github.io Code https://github.com/cambrian-mllm/cambrian-p Cambrian- P Models https://hf.co/collections/nyu-visionx/cambrian-p Data https://huggingface.co/datasets/nyu-visionx/Cambrian-P-Data Contents 1 Introduction Video is the projection of a dynamic 3D scene from a coherent sequence of viewpoints. Each viewpoint is defined by the observer’s pose, i.e. , its 3D position ( \mathbb{R}^{3} ) and orientation ( SO(3) ), specifying how the camera is embedded in the physical world 34 ; 61 . Pose provides the link between pixels and geometry, serving as a global anchor that relates distinct views to a shared coordinate frame 34 ; 78 . With pose, a video is no longer a collection of ambiguous image projections, but a coherent 3D scene 78 ; 85 . Recovering camera poses from visual observations has therefore been a cornerstone of 3D vision and robotics for decades. Structure from Motion (SfM) emerged in the 1980s precisely for this purpose 59 ; 78 , and remains a prerequisite for a wide array of downstream tasks: multi-view stereo 80 , 3D reconstruction, and neural rendering with NeRF 64 or 3D Gaussian Splatting 45 . In parallel, robotics and augmented-reality systems depend on SLAM 16 and visual odometry 67 to localize themselves and reason about their own motion. More recently, the community has recognized that jointly predicting pose and depth with feed-forward transformers can recover dense 3D structure in a single pass 96 ; 53 , further underscoring the centrality of pose as a geometric primitive. Yet, the role of pose has remained confined to 3D vision. We argue that its value extends far beyond. Multimodal large language models (MLLMs) 13 ; 91 ; 92 ; 5 ; 32 ; 1 ; 57 ; 89 now excel at semantic video understanding—recognizing actions, summarizing narratives, and answering questions—but consistently struggle when tasks demand spatial reasoning 88 ; 46 ; 108 ; 109 . We contend that this failure is not incidental. Without explicit grounding in 3D geometry, each frame is processed as an independent 2D snapshot, disconnected from the spatial structure across views. Our key insight is that pose naturally closes this gap. It is the lightest 3D signal, compactly encoding how views relate geometrically; it enforces global consistency through rigid-body (SE(3)) constraints; and it disentangles camera motion from scene dynamics, collapsing the space of plausible spatial interpretations. These properties also underpin human vision: viewers naturally separate their own motion from motion in the scene, and maintain a coherent 3D world across viewpoints. Together, they make pose not merely a useful auxiliary cue but a foundational inductive bias for video understanding. Figure 1 : Cambrian- P illustration in video QA . Cambrian- P equips the current video understanding paradigm with native camera pose prediction using an extra camera token per frame. Cambrian- P positions video frames into a shared spatial coordinate frame, then effectively models the underlying 3D world projected in video. Note that Cambrian- P only requires extra learnable pose tokens during training. In this work, we present Cambrian- P , which grounds pixels with camera poses by introducing pose supervision into MLLMs, calling for a new paradigm for video understanding. Cambrian- P introduces minimal architectural overhead: a single learnable camera token is appended to each frame’s visual features, and a lightweight projector and head regress pose parameters from the LLM’s hidden states. However, naively adding a pose regression loss does not work out of the box. Through careful investigation, we find that pose estimation and video question answering rely on fundamentally different frame sampling and data augmentation strategies. Uniform temporal sampling common in video understanding provides a shortcut for memorizing poses, while the heavy augmentation typical in pose estimation distracts from semantic comprehension. To reconcile these tensions, we design an interleaved training strategy that incorporates pose-only training data processed and augmented following its preferred practices, along with a random-jitter frame sampling strategy that introduces controlled perturbation to the conventional uniform sampling used for VQA. With this lightweight design and optimized joint training, Cambrian- P achieves state-of-the-art spatial VQA performance and competitive streaming camera pose estimation. On VSI-Bench 109 and VS Temporal I-Bench 29 , Cambrian- P obtains 4.5% \sim 6.5% gains over its no-pose counterpart, and demonstrates strong out-of-distribution generalization across eight spatial and general VQA benchmarks, including MindCube 117 , MVBench 50 , and EgoSchema 62 . Scaling pose supervision to in-the-wild videos with pseudo annotations further improves general video QA benchmarks, positioning pose as a promising signal for video understanding beyond spatial reasoning. For pose estimation, Cambrian- P delivers state-of-the-art streaming camera pose accuracy on ScanNet ATE, outperforming specialist reconstruction models such as StreamVGGT 130 , CUT3R 98 , and Point3R 103 . We further observe consistent scaling with model size, data size, and training iterations for both tasks. Finally, our analysis reveals two additional insights: (i) while adding depth supervision may seem intuitive for injecting 3D priors, it proves suboptimal for VQA compared to learning from camera pose; and (ii) camera pose enables MLLMs to think more globally across video frames, evidenced by the significant improvement of Cambrian- P when answering questions about spatially distant objects. 2 Related Work Multimodal Large Language Models Driven by the tremendous success of Large Language Models (LLMs) 13 ; 91 ; 92 ; 5 ; 32 ; 1 in linguistic understanding and reasoning, alongside powerful pretrained visual representations 72 ; 35 ; 68 ; 120 ; 93 , Multimodal Large Language Models (MLLMs) 47 ; 6 ; 49 extend LLMs beyond language-only corpora and have achieved impressive progress in understanding visual media such as images 56 ; 89 ; 47 ; 86 ; 24 ; 52 ; 22 and videos 111 ; 97 ; 7 ; 126 ; 74 ; 83 . However, despite their remarkable success in semantic parsing 23 ; 2 , world knowledge acquisition 118 ; 36 ; 77 , and general reasoning 60 ; 119 , MLLMs are still far from achieving human-level embodied intelligence capable of perceiving, reasoning, and acting within the