Mitigating Semantic Collapse in Partially Relevant Video Retrieval
WonJun Moon, MinSeok Jung, Gilhan Park, Tae-Young Kim Cheol-Ho Cho, Woojin Jun, Jae-Pil Heo Sungkyunkwan University {wjun0830, alstjr88, a01152a, jackdawson,gersys, junwoojin, jaepilheo}@skku.edu · 2025
Read Mitigating Semantic Collapse in Partially Relevant Video Retrieval by WonJun Moon, MinSeok Jung, Gilhan Park, Tae-Young Kim Cheol-Ho Cho, Woojin Jun, Jae-Pil Heo Sungkyunkwan University {wjun0830, alstjr88, a01152a, jackdawson,gersys, junwoojin, jaepilheo}@skku.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27432.txt.
Introduction
WonJun Moon, MinSeok Jung, and five other co-authors fix a flaw in video search in this 2025 paper. In partially relevant video retrieval, only part of a video matches the search text. Training treats each text-video pair as a match and all others as misses. That crude rule ignores how much videos and texts vary.
The flaw, which they call semantic collapse, squeezes distinct events in one video into the same corner of the model's map. At the same time, near-twin events in different videos get pushed apart. Their fix has two parts. One keeps the natural closeness among texts that the base model already knows. The other aligns video embeddings, the map points for video clips, across time scales, with token merging that keeps each segment coherent yet distinct. Tests show clear gains in retrieval quality.
Catalog record
| CREATOR | WonJun Moon, MinSeok Jung, Gilhan Park, Tae-Young Kim Cheol-Ho Cho, Woojin Jun, Jae-Pil Heo Sungkyunkwan University {wjun0830, alstjr88, a01152a, jackdawson,gersys, junwoojin, jaepilheo}@skku.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-27432
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Semantic Collapse in Partially Relevant Video Retrieval WonJun Moon, MinSeok Jung, Gilhan Park, Tae-Young Kim Cheol-Ho Cho, Woojin Jun, Jae-Pil Heo Sungkyunkwan University {wjun0830, alstjr88, a01152a, jackdawson,gersys, junwoojin, jaepilheo}@skku.edu Abstract Partially Relevant Video Retrieval (PRVR) seeks videos where only part of the content matches a text query. Existing methods treat every annotated text–video pair as a positive and all others as negatives, ignoring the rich semantic variation both within a single video and across different videos. Consequently, embeddings of both queries and their corresponding video-clip segments for distinct events within the same video collapse together, while embeddings of semantically similar queries and segments from different videos are driven apart. This limits retrieval performance when videos contain multiple, diverse events. This paper addresses the aforementioned problems, termed as semantic collapse, in both the text and video embedding spaces. We first introduce Text Correlation Preservation Learning, which preserves the semantic relationships encoded by the foundation model across text queries. To address collapse in video embeddings, we propose Cross-Branch Video Alignment (CBVA), a contrastive alignment method that disentangles hierarchical video representations across temporal scales. Subsequently, we introduce order-preserving token merging and adaptive CBVA to enhance alignment by producing video segments that are internally coherent yet mutually distinctive. Extensive experiments on PRVR benchmarks demonstrate that our framework effectively prevents semantic collapse and substantially improves retrieval accuracy. † † footnotetext: Equal contribution. 1 Introduction Figure 1 : Illustration of semantic collapse. ( Up ) Untrimmed videos in PRVR encompass diverse semantics that can be described by different texts. As a result, semantic segments (both text and video clips) from the same video may convey very different meanings, while segments from different videos can nonetheless be closely related. For example, Q2 of Video A and Q1 of Video B both depict “holding a dog”. ( Down ) Since all queries tied to a given video are treated as positives and negative queries drawn from other videos, the model pulls together all text embeddings (and their corresponding video segments) for that video, regardless of true meaning, and pushes apart semantically similar queries (and segments) from different videos. (a) illustrates that queries of the same video are pulled together regardless of their semantic relationships (left), while queries with similar context (holding a dog) are pushed apart (right). (b) shows that video segments also suffer from the same phenomenon. Recently, Partially Relevant Video Retrieval (PRVR) ( ms-sl_PRVR ; gmm_PRVR ; gmmv2_PRVR ) has emerged as a significant research challenge in computer vision. PRVR shares the same objective as traditional Text-to-Video Retrieval Alea_VR ; CLIPPING_VR ; MMCKD_VR ; Pidro_VR ; VoP_VR ; X-clip_VR , retrieving the video that best aligns with a given text query. However, the key difference lies in PRVR’s assumption that target videos may be only partially relevant to the query rather than requiring a perfect semantic match. The primary challenge in PRVR lies in learning from text-video pairwise annotations. A single video is often associated with multiple distinct text queries labeled as positive pairs; however, the semantic relationships among these text queries are not explicitly defined, and fine-grained temporal annotations that indicate their precise alignment within the video are typically unavailable. As a result, conventional training for retrieval based on the InfoNCE loss infonce_Loss ; supercon induces a semantic collapse problem in PRVR. Semantic collapse refers to the phenomenon where paired text queries and visual segments are excessively attracted to each other while being indiscriminately repelled from features of other pairs, regardless of their actual semantic similarity. Fig. 1 (a) illustrates this issue within the text embedding space; text queries associated with the same video tend to cluster together even when they are semantically unrelated, while semantically similar queries are pulled apart when they are paired with different videos. In addition, the same phenomenon occurs in video embeddings; video segments drawn from the same video collapse together regardless of their true semantic differences, as shown in Fig. 1 (b). This is because the training guidance is provided by video ID, not by their individual semantic content. In short, every segment in a video shares the identical set of paired text queries as positives. Previous works, e.g., GMMFormer gmm_PRVR and GMMFormer-v2 gmmv2_PRVR , have attempted to address the semantic collapse within text embeddings. Specifically, these methods explicitly reduce the similarity between text queries paired with the same video. However, the semantic relationships between text queries are often overlooked, and the issue of semantic collapse within video embeddings remains underexplored, leading to sub-optimal performance. In this paper, we aim to mitigate the semantic collapse in both text and video embeddings for PRVR. First, we introduce Text Correlation Preservation Learning (TCPL), which leverages CLIP clip_ID , a vision-language foundation model with a well-structured semantic space. By distilling the semantic relationships encoded in CLIP, TCPL effectively regularizes the semantic collapse within text embeddings. While TCPL leverages CLIP’s rich text-semantic structure to regularize collapse in the textual embedding space, we point out that the same approach cannot be directly applied to video embeddings. This is because CLIP’s pretraining operates on static images, thereby lacking the capacity to model temporal dynamics li20254d . To this end, we introduce Cross-Branch Video Alignment (CBVA), a dedicated objective to preserve context diversity in the video modality. CBVA utilizes a dual-branch architecture commonly adopted in PRVR to encode hierarchical video representations and employs a contrastive objective to differentiate distinct events within a video. Concretely, frame- and clip-level embeddings from the same timestamp are encouraged to align closely, while those from different timestamps are driven apart. Then, we further leverage the token merging strategy in two ways to enhance video-adaptivity within CBVA; (1) order-preserving token merging is introduced for semantically consistent video clip aggregation, and (2) bipartite token merging tome is leveraged to organize representative contexts within each video. By encoding clips in a context-aware manner, we encourage videos to be represented in line with their true semantic content. Consequently, with TCPL and CBVA combined, our method achieves state-of-the-art performances in all tested benchmarks. In summary, our contributions are (1) We propose Text Correlation Preservation Learning, which leverages the semantic relationships within the foundation model to address semantic collapse within text embeddings, (2) We propose Cross-Branch Video Alignment to mitigate the semantic collapse in video modality by distinguishing distinct events within a video, (3) We leverage token merging strategies to encourage the precise video alignment, and (4) Our method achieves superior performances across all datasets in PRVR. 2 Related Work Partially Relevant Video Retrieval. PRVR aims to retrieve untrimmed videos that are partially relevant to a given query ms-sl_PRVR ; pean ; jun2025bridging ; mgakd . MS-SL ms-sl_PRVR addresses this challenge by proposing a dual encoding strategy that explicitly separates features for frame and clip segments, capturing different temporal scales within untrimmed videos. Subsequently, DL-DKD dldkd_PRVR leverages CLIP clip_ID to enhance PRVR performance by distilling text–frame similarity. GMMFormer gmm_PRVR introduces a Gaussian Mixture Model–based Transformer that enables efficient retrieval with a reduced set of video features. It also identifies semantic collapse as a key challenge and proposes a query-diverse loss to enforce separation among multiple text queries linked to the same video. Building on this, GMMFormer v2 gmmv2_PRVR further addresses semantic collapse by explicitly controlling the