The Devil is in the Spurious Correlations: Boosting Moment Retrieval with Dynamic Learning
Xinyang Zhou, Fanyue Wei, Lixin Duan, Angela Yao, Wen Li · 2025
Read The Devil is in the Spurious Correlations: Boosting Moment Retrieval with Dynamic Learning by Xinyang Zhou, Fanyue Wei, Lixin Duan, Angela Yao, Wen Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-07305.txt.
Introduction
Xinyang Zhou and co-authors fix a subtle failure in systems that find the moment a text query describes inside a video. Their 2025 paper shows that models often cheat: they link queries to background scenes instead of the true moment, because the two so often appear together. That shortcut blurs the exact start and end times the task demands.
Two remedies follow. New synthetic videos place the target moment against shifting backgrounds, forcing the model to watch the moment itself. A text-dynamics step then aligns words with changing frames rather than static scenes. On two standard benchmarks the method sets new best scores, and tests across designs show the tricks transfer well.
Catalog record
| CREATOR | Xinyang Zhou, Fanyue Wei, Lixin Duan, Angela Yao, Wen Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-07305
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Devil is in the Spurious Correlations: Boosting Moment Retrieval with Dynamic Learning Xinyang Zhou, Fanyue Wei, Lixin Duan, Angela Yao, Wen Li Abstract Given a textual query along with a corresponding video, the objective of moment retrieval aims to localize the moments relevant to the query within the video. While commendable results have been demonstrated by existing transformer-based approaches, predicting the accurate temporal span of the target moment is still a major challenge. This paper reveals that a crucial reason stems from the spurious correlation between the text query and the moment context. Namely, the model makes predictions by overly associating queries with background frames rather than distinguishing target moments. To address this issue, we propose a dynamic learning approach for moment retrieval, where two strategies are designed to mitigate the spurious correlation. First, we introduce a novel video synthesis approach to construct a dynamic context for the queried moment, enabling the model to attend to the target moment of the corresponding query across dynamic backgrounds. Second, to alleviate the over-association with backgrounds, we enhance representations temporally by incorporating text-dynamics interaction, which encourages the model to align text with target moments through complementary dynamic representations. With the proposed method, our model significantly alleviates the spurious correlation issue in moment retrieval and establishes new state-of-the-art performance on two popular benchmarks, i.e . , QVHighlights and Charades-STA. In addition, detailed ablation studies and evaluations across different architectures demonstrate the generalization and effectiveness of the proposed strategies. Our code will be publicly available. † † footnotetext: ⋆ Equal contribution. 1 Introduction Videos are a ubiquitous media format, but it can be time-consuming to browse through videos to localize specific moments. Using text to retrieve corresponding moments within a lengthy video [ 50 , 2 ] plays an important role for entertainment, search, etc . Existing moment retrieval approaches [ 18 , 26 , 45 , 16 , 15 ] leverage the DETR [ 4 ] detection transformer architecture to fuse text query into video representation. They show remarkable results in detecting the target moment by text-video alignment, but accurately predicting the temporal span remains a significant challenge. The learned video representations [ 18 , 16 ] are aligned with the semantics of the text, thus generally biased to subjects or spatial objects. Consequently, the semantic elements in background frames are overly associated with textual queries. Such inappropriate association between irrelevant contexts and labels is the manifestation of Spurious correlation [ 1 , 39 ] , which are under-explored in moment retrieval tasks. This work identifies such spurious correlations between the text queries and moment visual context as the root cause for this performance gap in moment retrieval. Specifically, learned SOTA moment retrievers [ 16 , 26 , 18 ] tend to associate text queries with the background frames rather than distinguishing the target moment. Figure illustrates how such spurious issues lead to a sub-optimal performance for moment retrieval. For example, the SOTA method, BAM-DETR [ 16 ] , predicts a similar temporal span for an ‘The blonde woman works out in a gym with red lighting.’ in both the original video and an altered version where the target moment is masked out. In the masked case, the segment near the mask is a more reasonable output but the baseline methods [ 26 , 16 , 45 , 25 ] predict the masked segment from background frames with the learned spurious correlation, which is the most unrelated output to the text query. To address the spurious issues, we propose a dynamic learning method with two novel strategies to mitigate spurious correlation. First, we propose a synthesis strategy that dynamically contextualizes the target moments for retrieval. We select similar video pairs and augment the samples by synthesizing target moments within similar contexts. This encourages the model to distinguish target moments from queries across more dynamic contexts, alleviating the over-association with background frames. Second, we propose to enhance video-text representations to reduce spurious dependencies on background frames by aligning text queries with temporal dynamics. To achieve this, we design a simple yet efficient temporal tokenizer to extract dynamic representations. Additionally, a text-dynamics interaction module is introduced to align dynamics with text queries, enabling the model to establish a stand-up correlation with complementary dynamic representations while mitigating background bias. Experiments on two challenging benchmarks, i.e . , QVHighlights and Charades-STA, show that our method outperforms existing methods by clear margins. Besides, spurious correlation in moment retrieval is validated to be largely alleviated by spurious evaluation. The detailed ablation analyses demonstrate the effectiveness of the proposed strategies to resolve the spurious issues. In conclusion, our contributions to this work are summarized as follows: • To the best of our knowledge, we are the first to investigate the spurious correlation in moment retrieval. • We propose a dynamic learning framework for moment retrieval that mitigates spurious correlations by dynamically contextualizing target moments through novel video synthesis and enhancing representations with text-dynamics alignment. • The proposed method achieves SOTA performance across all benchmarks. Besides, our model also provides a strong interpretation of alleviating spurious correlations . 2 Related Work 2.1 Moment Retrieval Moment Retrieval aims to predict a temporal span within a video corresponding to a given natural language query [ 50 ] . Early methods had two stages: first, sampling candidate moments as proposals, and then scoring the proposals to obtain the final predictions [ 8 , 2 , 22 , 21 , 9 ] . However, more recent approaches are one-shot [ 48 , 46 , 52 , 47 , 55 , 17 , 33 , 43 ] to predict the temporal span directly. Simultaneously, hybrid frameworks that integrate transformers for feature learning and fusion with CNNs for prediction have been widely studied. SnAG [ 27 ] explored the impact of cross-modal fusion on the scalability of video grounding models and proposed an efficient training scheme. Although many moment retrieval methods have achieved remarkable performance, the majority primarily focus on architectural design, such as exploring new networks [ 48 , 46 ] , enhancing video-text alignment [ 17 ] , or refining attention mechanisms [ 27 ] . Generally, these methods focus on how to fuse text and video representations to improve the temporal span prediction but the overall performance are limited by the inefficient alignment architecture. Figure 1 : Overview of the proposed TD-DETR . Video pairs are sampled by their similarity and then forwarded into the Video Synthesizer to generate Dynamic Context (Section 3.1 ), Then, the Dynamic Context, i.e . , synthesized pairs, are enhanced by Temporal Dynamic Tokenization and interact with textual information (Section 3.2 ). Finally, the enhanced video tokens are sent to the transformer encoder-decoder with prediction heads to predict moments (Section 3.3 ). 2.2 Detection Transformers In the past several years, the wide adoption of detection transformers (DETR) [ 4 ] in video moment retrieval has gained remarkable performance [ 14 , 18 , 23 , 26 , 16 , 45 , 44 ] . While introducing the DETR framework to the task of moment retrieval, it demonstrated the effectiveness on QVHighlights dataset [ 18 ] . Furthermore, DETR-based approaches have explored the relationship between moment retrieval and highlight detection,