LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization

Zirui Shang, Xinxiao Wu, Shuo Yang · 2025

Read LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization by Zirui Shang, Xinxiao Wu, Shuo Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24282.txt.

Introduction

Zirui Shang, Xinxiao Wu, and Shuo Yang improve how models find action edges in videos from a text query. Their 2025 paper notes that a query tells what an action is, but rarely how it starts or ends. Hand labels for those edges thus carry much guesswork. That noise can destabilize training.

Their fix uses large language models to write extra text that describes the start and end of the action. Those cues guide the search for edges. For training, they score edges by how well frames match the expanded query and by time distance to labeled frames. That gives steadier supervision. The method plugs into existing models with no redesign. Tests on several data sets show clear gains.

Catalog record

CREATORZirui Shang, Xinxiao Wu, Shuo Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24282

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM-powered Query Expansion for Enhancing Boundary Prediction in Language-driven Action Localization Zirui Shang, Xinxiao Wu, Shuo Yang Abstract Language-driven action localization in videos requires not only semantic alignment between language query and video segment, but also prediction of action boundaries. However, the language query primarily describes the main content of an action and usually lacks specific details of action start and end boundaries, which increases the subjectivity of manual boundary annotation and leads to boundary uncertainty in training data. In this paper, on one hand, we propose to expand the original query by generating textual descriptions of the action start and end boundaries through LLMs, which can provide more detailed boundary cues for localization and thus reduce the impact of boundary uncertainty. On the other hand, to enhance the tolerance to boundary uncertainty during training, we propose to model probability scores of action boundaries by calculating the semantic similarities between frames and the expanded query as well as the temporal distances between frames and the annotated boundary frames. They can provide more consistent boundary supervision, thus improving the stability of training. Our method is model-agnostic and can be seamlessly and easily integrated into any existing models of language-driven action localization in an off-the-shelf manner. Experimental results on several datasets demonstrate the effectiveness of our method. keywords LLM-powered Query Expansion, Enhancing Boundary Prediction, Language-driven Action Localization 1 Introduction Language-driven action localization, also known as video moment retrieval Gao and Xu 2021 ; Wu et al. 2021 ; Zeng et al. 2021 ; Tang et al. 2021 or temporal sentence grounding Yuan et al. 2019 ; Liu et al. 2021b ; Liu et al. 2021c , aims to localize the temporal action segment semantically aligned with a natural language query in an untrimmed video. It has received increasing attention in recent years and has wide applications in many downstream tasks such as video question answering Sun et al. 2021 and video editing Zhang et al. 2022b . Recent studies Gao et al. 2017 ; Liu and Zhou 2023 ; Liu et al. 2021a ; Zhang et al. 2020b have achieved remarkable progress in language-driven action localization by aligning the video segment with the language query Ghosh et al. 2019 ; Zhang et al. 2023 ; Liu et al. 2022a ; Liu et al. 2024 and predicting the desired action boundaries using rigid boundary annotations as supervision Zhang et al. 2020a ; Liu and Hu 2022 ; Liu et al. 2022b ; Liu et al. 2021b . Figure 1: Illustration of boundary uncertainty for the same language query. The orange text represents the motion of the start boundary annotation and the blue text represents the motion of the end boundary annotation. However, the language query primarily describes the main content of an action, and lacks details of start and end boundaries. Different human annotators may have different subjective interpretations of action boundaries, and give inconsistent boundary annotations for the same type of action in different videos, resulting in boundary uncertainty. As shown in Figure 1 , for the language query of “person eats sandwich” that describes the same action “eat”, the start boundary annotation of one video (a) refers to the motion “place hand on food plate”, which that of another video (b) refers to the motion “put food into the mouth”. Therefore, learning to localize accurate boundary timestamps from inconsistent boundary annotations is highly challenging and is still under exploration. To address this challenge, we propose LLM-powered query expansion, which leverages a large language model (LLM) to generate detailed textual descriptions of action start and end boundaries to expand the original query. Specifically, we first design several prompts, such as “Please describe the beginning and ending process in one sentence of the following action [ACTION], The description you generate cannot contain any objects that are not present in the action” , for the LLM to generate fine-grained descriptions about motions, poses, or temporal differences that distinguish boundaries. Then, we propose a query-guided temporal modeling module that incorporates the expanded query into visual representation learning, which benefits perception of crucial temporal changes in actions for recognition. The expanded query provides explicate boundary information, which helps achieve unambiguous modeling and reduce the impact of uncertainty. Furthermore, to enhance the tolerance to boundary uncertainty during training, we propose a boundary probability modeling module that estimates the probability of each video frame being an action boundary based on the expanded query. Specifically, for each frame, we model the probability of being an action boundary by calculating its semantic similarity to the expanded query and its temporal distance to the labeled boundary frame. The semantic similarity ensures that frames relevant to the action described by the expanded query are prioritized, while the temporal distance reflects the intuition that frames closer to the annotated boundary are more likely to correspond to the action boundary. The probability scores, instead of rigid annotations, reduce over-reliance on a single annotated boundary by providing more flexible and reliable supervision, thereby better capturing motion-related information near the boundary. In addition, the clear guidance of the expanded query ensures that the probability scores are more consistent across similar actions, further providing more consistent supervision and improving the stability of training. The main contributions can be summarized as follows: • We propose a query expansion method that expands the language query through LLM to capture more details of action boundaries, effectively reducing the impact of boundary uncertainty in action localization. • We propose a boundary probability modeling module that utilizes the expanded query to transform rigid boundary annotations into probability scores for training, successfully improving the robustness to boundary uncertainty. • Our method can be seamlessly and easily integrated into existing models in an off-the-shelf manner. Extensive experiments on five state-of-the-art models across three benchmark datasets demonstrate the effectiveness of our method. 2 Related Work Existing methods of language-driven action localization can be broadly categorized into two groups: proposal-based methods and proposal-free methods. The proposal-based methods first generate segment proposals using sliding windows, proposal generation, or anchor-based methods and then rank them according to the query. Through exhausting proposal generation and directly aligning the semantics of the query and proposals, the proposal-based methods achieve better performance. Early methods such as CTRL Gao et al. 2017 and MCN Hendricks et al. 2017 use sliding windows of varying scales to generate candidate video segments, while later methods such as SPN Xu et al. 2017 , QSPN Xu et al. 2019 , and SAP Chen and Jiang 2019 adopt segmented proposal networks tailored to specific queries. Recent studies, including 2D-TAN Zhang et al. 2020b , RaNet Gao et al. 2021 , and CDN Wang et al. 2022 , utilize two-dimensional feature maps to model relationships across segments of different durations. MIPGN Fang et al. 2025 proposes a multi-modal integrated proposal generation network to guide adaptive proposal generation, incorporates object and motion tags for richer multi-modal representations, and employs dynamic negative proposal mining to improve retrieval accuracy. VGCI Lv and Su 2025 introduces a variational inference framework for weakly supervised video moment retrieval, modeling a global clue as a latent Gaussian variable to guide query reconstruction and enhance video-query alignment. The proposal-free methods first learn cross-modal interactions between query

Read the complete text