Who Can We Trust? Scope-Aware Video Moment Retrieval with Multi-Agent Conflict

Chaochen Wu, Guan Luo, Meiyun Zuo, Zhitao Fan · 2025

Read Who Can We Trust? Scope-Aware Video Moment Retrieval with Multi-Agent Conflict by Chaochen Wu, Guan Luo, Meiyun Zuo, Zhitao Fan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00370.txt.

Introduction

Chaochen Wu, Guan Luo, Meiyun Zuo, and Zhitao Fan work on video moment retrieval: given a text query, find the matching segment inside a long, untrimmed video. Combining several models should help, but models often disagree on the segment boundaries, and earlier systems had no way to settle such conflicts. Their model scans the whole video once, returns boundaries, and produces evidence for its choice.

On top sits a multi-agent framework where agents argue their locations, and evidential learning weighs the conflict. A useful side effect follows: the system can declare a query out of scope, with no matching moment in the video, without extra training. Benchmark tests beat leading rivals. The authors argue that modeling conflict between agents is itself a strong way to lift retrieval performance.

Catalog record

CREATORChaochen Wu, Guan Luo, Meiyun Zuo, Zhitao Fan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00370

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Who Can We Trust? Scope-Aware Video Moment Retrieval with Multi-Agent Conflict Chaochen Wu, Guan Luo, Meiyun Zuo, Zhitao Fan Abstract Video moment retrieval uses a text query to locate a moment from a given untrimmed video reference. Locating corresponding video moments with text queries helps people interact with videos efficiently. Current solutions for this task have not considered conflict within location results from different models, so various models cannot integrate correctly to produce better results. This study introduces a reinforcement learning-based video moment retrieval model that can scan the whole video once to find the moment’s boundary while producing its locational evidence. Moreover, we proposed a multi-agent system framework that can use evidential learning to resolve conflicts between agents’ localization output. As a side product of observing and dealing with conflicts between agents, we can decide whether a query has no corresponding moment in a video (out-of-scope) without additional training, which is suitable for real-world applications. Extensive experiments on benchmark datasets show the effectiveness of our proposed methods compared with state-of-the-art approaches. Furthermore, the results of our study reveal that modeling competition and conflict of the multi-agent system is an effective way to improve RL performance in moment retrieval and show the new role of evidential learning in the multi-agent framework. Introduction In recent years, the development of visual recording technologies has created massive amounts of video data. However, it is time-consuming for people to locate moments they are interested in from long, untrimmed videos. Video moment retrieval(MR), also known as temporal grounding, uses text sentences to retrieve moments from videos, which brings convenience for searching information from untrimmed videos. Examples of applications of MR include finding desired scenes from movies, locating patients’ dangerous accidents from hospital surveillance videos, and evaluating athletes’ performance from sports game videos. Given a text query and a video reference, the cross-modal MR model should output a start timestamp and an end timestamp to indicate the corresponding moment in video reference, and we show an example in Figure 1 . In the MR task, the MR model takes one query sentence and one video reference as input. For the query “person starts cooking with a pan.”, the model outputs its corresponding timestamps (starting from 9.2 seconds and ending in 18.1 seconds). For the query “the person closes the laptop.”, the video reference does not have any corresponding moment, so it is an out-of-scope reference which does not have valid output. Figure 1: An example of video moment retrieval. One of the solutions for MR is a mode like the ranking task, which is also called anchor-based methods ( Gao et al. 2017 ; Hendricks et al. 2018 ; Liu et al. 2018 ; Chen and Jiang 2019 ; Xu et al. 2019 ; Zhang et al. 2019 ) : First, it cut video reference in pre-defined window sizes and produced video segments; Then, it encoding video segments and query to features; Finally, it compute similarity between the query feature and video segments features and find the fittest video segments. Another type of solution called anchor-free approaches ( Chen et al. 2018 ; Yuan, Mei, and Zhu 2019 ; Lu et al. 2019 ; Chen et al. 2020 ; Zhang et al. 2020a ; Cao et al. 2021 ) , which using query features and video features to predict boundaries of video moments within all video frames. Most previous efforts only consider that videos are fully relevant to the query, but sometimes queries are not relevant to the video reference (out-of-scope). Some studies ( Paul, Mithun, and Roy-Chowdhury 2021 ; Sun et al. 2021 ; Chen et al. 2023 ) study this issue with joint searching and grounding solutions. However, pursuing success in the search task requires the model to take extra training that needs training datasets for searching and may negatively affect the models’ MR capacity. Besides the methods we already mentioned, some methods are built with reinforcement learning (RL). SM-RL ( Wang, Huang, and Wang 2019 ) used a recurrent neural network-based agent to observe a sequence of video frames dynamically and finally outputs the temporal boundaries of the given text query. Mimic a person watching the video, the RL agent can move the moment’s boundary driven by rewards ( He et al. 2019 ) , and in this study, the reward is computed by Intersection over Union (IoU) change: if an action increases IoU of the agent’s current location and the objective location, the model will get a positive reward value. Otherwise, it will have a penalty. An improved method, MABAN ( Sun, Wang, and He 2021 ) , uses multi-agent RL to retrieve video moment: it uses one agent to move the start boundary and one agent for the end, and two agents share one observation network and environment. RewardTLG ( Zeng, Pan, and Han 2023 ) uses a vision-language pre-training model as a reward model, which provides flexible rewards different from previous RL methods. Figure 2: Method overview. The left part is the action and output for ESRL. The right part is the architecture of MARLCC. ESRL produces IoU, location, and evidence for each step, and it moves in the right direction in a fixed window size and step size. For MARLCC, ESRL is one of the agents from three. MARLCC uses agents’ trusted IoU to select the best result from competition , and in the 2DSTB map, it is the “closest” agent to the ground truth; MARLCC also can use conflict to find OOS query, and in the 2DSTB map, the OOS query has higher conflict than matched queries. Applying multi-agent cooperation to MR has these advantages: it decouples the start boundary movement with end boundary movement, which helps agents to find objective boundaries with higher flexibility; if one agent takes the wrong action and others take the correct actions, the final result may still approach to the objective moment’s location. However, multi-agent cooperation has these limitations: the success of getting final moments on the basis of different agents collaborating properly, which brings extra complexity within the model design; the goal of cooperation restricts the designation of agents because we have to design agents’ action types and reward functions carefully; if we want collaborated agents to produce an accurate result, each model have to perform well, but it is a challenging goal for black-box neural network models. In this study, we propose a novel multi-agent system (MAS) framework for MR that sheds light on competition and conflict within agents. First, we incorporate evidential learning into RL-based MR models, and when RL agents take their actions, it will produce evidence and uncertainty for its relative boundary locations; second, inspired by previous MR methods that clip the video to segments, we build an RL-based MR model that scans all video feature with a fix window size and moves towards one direction; third, our method can produce agents’ trusted IoU, which helps us to select the best localization output among different agents ( competition ); fourth, various agents take their action independently, so we can evaluate the conflict of their final locations to find out-of-scope (OOS) queries in a zero-shot manner. In real-world applications, users may give some queries that are beyond the

Read the complete text