Dynamic Reflections: Probing Video Representations with Text Alignment

Tyler Zhu, Tengda Han, Leonidas Guibas, Viorica Pătrăucean, Maks Ovsjanikov, Google DeepMind · 2025

Read Dynamic Reflections: Probing Video Representations with Text Alignment by Tyler Zhu, Tengda Han, Leonidas Guibas, Viorica Pătrăucean, Maks Ovsjanikov, Google DeepMind (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02767.txt.

Introduction

Tyler Zhu and four colleagues run the first broad study of how video and text representations line up. Alignment between image and text encoders has revealed much about those models, but video adds time, which earlier work mostly skipped. Their 2025 study probes modern video and language encoders side by side.

Alignment strength depended heavily on how rich the test inputs were, both in frames and in captions. The team fit parametric test-time scaling laws that predicted the observed behavior well. Strong alignment with text also tracked with better scores on semantic and some non-semantic tasks, hinting at general-purpose video understanding. Temporal reasoning tests built on alignment gave the models a hard challenge. Video-text alignment, they conclude, is a cheap zero-shot probe of encoder power.

Catalog record

CREATORTyler Zhu, Tengda Han, Leonidas Guibas, Viorica Pătrăucean, Maks Ovsjanikov, Google DeepMind
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02767

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamic Reflections: Probing Video Representations with Text Alignment Tyler Zhu, Tengda Han, Leonidas Guibas, Viorica Pătrăucean, Maks Ovsjanikov, Google DeepMind Abstract The alignment of representations from different modalities has recently been shown to provide insights on the structural similarities and downstream capabilities of different encoders across diverse data types. While significant progress has been made in aligning images with text, the temporal nature of video data remains largely unexplored in this context. In this work, we conduct the first comprehensive study of video-text representation alignment, probing the capabilities of modern video and language encoders. Our findings reveal several key insights. First, we demonstrate that cross-modal alignment highly depends on the richness of both visual (static images vs. multi-frame videos) and text (single caption vs. a collection) data provided at test time , especially when using state-of-the-art video encoders. We propose parametric test-time scaling laws that capture this behavior and show remarkable predictive power against empirical observations. Secondly, we investigate the correlation between semantic alignment and performance on both semantic and non-semantic downstream tasks, providing initial evidence that strong alignment against text encoders may be linked to general-purpose video representation and understanding. Finally, we correlate temporal reasoning with cross-modal alignment providing a challenging test-bed for vision and language models. Overall, our work introduces video-text alignment as an informative zero-shot way to probe the representation power of different encoders for spatio-temporal data. 1 Introduction A hallmark of general intelligence is the ability to reason about the world in its different manifestations and to integrate different sensory data into unified mental representations ( Ernst & Bülthoff 2004 ; Kudithipudi et al. 2022 ) . The pursuit of such unified representations is also fundamental to the development of capable multimodal AI systems and for the advancement of embodied agents that must operate within a complex world ( Xie et al. 2024 ; Fung et al. 2025 ) . In recent years, significant progress has been made towards this goal. The success of multimodal learning approaches has highlighted the potential of joint training paradigms ( Radford et al. 2021 ; Yin et al. 2024 ; Liu et al. 2024b ; Dubey et al. 2024 ; Team et al. 2023 ; Team et al. 2025 ) , while a parallel line of research has provided evidence that even unimodal models, when trained at scale, develop representations with strong structural similarities ( Huh et al. 2024 ; Maniparambil et al. 2024a ; Tjandrasuwita et al. 2025 ) . This observation has given rise to the Platonic Representation Hypothesis (PRH), which posits that neural networks converge towards a shared statistical model of reality in their latent spaces ( Huh et al. 2024 ) . Building on this, several recent works have established a wide range of techniques to evaluate and exploit cross-modal alignment, in both zero-shot and few-shot scenarios ( Maniparambil et al. 2024a ; Jha et al. 2025 ; Schnaus et al. 2025 ; Hadgi et al. 2025 ) . However, prior work has focused almost exclusively on the alignment of static modalities: primarily images and text. In other words, although the Platonic Representation Hypothesis has been stated in full generality, so far, its validity has only been evaluated on static data, leaving its applicability to dynamic, temporal domains an open question. Consequently, the rich information contained in video: motion, causality and temporal dependencies, exhibited, e.g., in intricate human interactions ( Gu et al. 2018 ; Wang et al. 2023 ) , has been largely overlooked in the context of representation alignment. A notable limitation of previous studies focusing on static modalities was raised in Huh et al. 2024 , where the authors pointed out: “the maximum theoretical value for the alignment metric is 1. Is a score of 0.16 indicative of strong alignment […] or does it signify poor alignment with major differences left to explain? We leave this as an open question. ” In this work, we provide a partial answer to this open question by showing that the limited alignment observed previously is, in large part, due to the impoverished data given at test time . Specifically, we demonstrate that by considering multiple video frames instead of a single image, as well as a diverse set of captions instead of a single annotation, the alignment score can be improved significantly, achieving close to 0.4 in some cases, without modifying the underlying trained models. This result highlights, for the first time, that large improvements in alignment can be achieved through efforts at test time , which is complementary to the training-time resources (model size, amount of training data, etc.) considered in prior work. It also suggests that multi-frame approaches can capture different aspects of a scene better, and points to the possibility of a strong zero-shot evaluation metric capable of probing the representation power of both image and video encoders. More broadly, in this work we extend previous cross-modal alignment studies into the temporal domain by conducting the first comprehensive investigation of video-text representation similarity. We introduce a robust evaluation framework to probe and compare the capabilities of 121 modern video and language models. Our findings reveal that the rich, temporal nature of video provides a powerful signal for semantic grounding. Specifically, we demonstrate that (1) state-of-the-art self-supervised video encoders, e.g., VideoMAEv2 ( Wang et al. 2023 ) , achieve competitive alignment with text compared to top-performing image encoders, e.g., DINOv2 ( Oquab et al. 2023 ) ; (2) we observe that alignment quality is highly sensitive to the richness of the provided visual and textual data, and (3) there is a non-trivial relation between the video-text alignment and the performance of video models on downstream tasks. Finally, we highlight challenging situations with hard negative examples, in which current video and text models struggle, leaving room for future improvement. 1 1 1 Our project page with code is at https://video-prh.github.io . 2 Related Work Our work is situated at the intersection of three domains: the study of emergent representation alignment, self-supervised video understanding, and multimodal learning. We first review the recent work in representation convergence, then discuss the most prominent paradigms for learning from video data, and finally, contextualize our work by highlighting the specific gap in probing the alignment of unimodal trained video models. The Platonic Representation Hypothesis and Emergent Alignment in Static Modalities A foundational concept for our investigation is the Platonic Representation Hypothesis (PRH), introduced by Huh et al. 2024 . The PRH posits that as neural networks are scaled in terms of capacity, data diversity, and task variety, their learned internal representations converge toward a shared, universal statistical model of reality. This converged latent structure is termed the “Platonic representation,” drawing an analogy to Plato’s notion of an ideal reality that underlies our sensory observations. This hypothesis has been empirically tested by measuring the geometric similarity of representation spaces, obtained using a wide range of encoders. The primary method for this is comparing the representational kernels, which characterize how models measure dissimilarity between data points ( Kornblith et al. 2019 ; Maniparambil et al. 2024b ; Huh et al. 2024 ) . The key finding supporting the PRH is that as unimodal vision and language models become more capable, the structure

Read the complete text