Towards Fine-Grained Human Motion Video Captioning Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/37460
Guorui Song, Guocun Wang, Zhe Huang, Jing Lin, Xuefei Zhe, Jian Li, Haoqian Wang · 2025
Read Towards Fine-Grained Human Motion Video Captioning Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/37460 by Guorui Song, Guocun Wang, Zhe Huang, Jing Lin, Xuefei Zhe, Jian Li, Haoqian Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24767.txt.
Introduction
Guorui Song, Guocun Wang, Zhe Huang, Jing Lin, Xuefei Zhe, Jian Li, and Haoqian Wang aim to make video captions name fine moves of the human body well. Past caption tools often give vague lines or lines that clash with the clip. Their model adds body mesh based move traits right into the word choice step.
Those move traits stress how the body shifts, which cuts fake claims and lines up words with the clip. To train and test such work, they built a set of more than one hundred thousand clip and line pairs on human moves. A bench set goes with it. Their model beat past tools on hard move clips.
Catalog record
| CREATOR | Guorui Song, Guocun Wang, Zhe Huang, Jing Lin, Xuefei Zhe, Jian Li, Haoqian Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24767
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Fine-Grained Human Motion Video Captioning Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3754560 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Video summarization CCS: Computing methodologies Natural language generation Guorui Song, Guocun Wang, Zhe Huang, Jing Lin, Xuefei Zhe, Jian Li, Haoqian Wang Abstract . Generating accurate descriptions of human actions in videos remains a challenging task for video captioning models. Existing approaches often struggle to capture fine-grained motion details, resulting in vague or semantically inconsistent captions. In this work, we introduce the Motion-Augmented Caption Model (M-ACM), a novel generative framework that enhances caption quality by incorporating motion-aware decoding. At its core, M-ACM leverages motion representations derived from human mesh recovery to explicitly highlight human body dynamics, thereby reducing hallucinations and improving both semantic fidelity and spatial alignment in the generated captions. To support research in this area, we present the Human Motion Insight (HMI) Dataset, comprising 115K video-description pairs focused on human movement, along with HMI-Bench, a dedicated benchmark for evaluating motion-focused video captioning. Experimental results demonstrate that M-ACM significantly outperforms previous methods in accurately describing complex human motions and subtle temporal variations, setting a new standard for motion-centric video captioning. † † cc-license: by-nc-sa 1. Introduction The rapid advancement of vision-language model has significantly improved video understanding capabilities across various applications. However, accurately capturing and describing human motion in videos remains a substantial challenge. Current state-of-the-art multimodal models ( Liu et al. 2024 ; Chen et al. 2024d ; Li et al. 2023 ; Li et al. 2024b ; Li et al. 2024a ; Wang et al. 2024 ; Bai et al. 2023 ; Maaz et al. 2024 ; Zhang et al. 2023 ; Lin et al. 2024 ) often struggle with precisely describing human movements, frequently generating hallucinations, providing overly generic descriptions, or incorrectly identifying body parts and actions. This limitation is particularly evident when models are tasked with describing fine-grained movements in videos, such as human body movements, subtle facial expressions, and other nuanced physical actions. Existing open-source multimodal large language models (e.g., Video-LLaVA ( Lin et al. 2023b ) , LLaVA-OV ( Li et al. 2024a ) , Qwen2VL ( Wang et al. 2024 ) ) demonstrate notably poor performance in human motion captioning tasks. They tend to produce vague or inaccurate answers and struggle to capture the fine-grained details of human body movements. For instance, when presented with a yoga video, these models might simply state that "a person is doing yoga" without describing the specific pose, body positioning, or movement. In the domain of motion-to-text conversion, most approaches rely on simplistic text combination methods without effectively leveraging the capabilities of large language models, resulting in limited generalization ability. Even specialized motion models ( Árbol and Casas 2024 ; Bugliarello et al. 2025 ; Pang et al. 2025 ; Wang et al. 2025a ; Zhao et al. 2025 ; Chen et al. 2024b ) integrated with large language models, such as MotionLLM ( Chen et al. 2024b ) , exhibit significant limitations when applied to video tasks. These models typically generate brief, overgeneralized responses and still suffer from severe hallucination issues. They demonstrate weak perception of micro-expressions, background information, and task states. Moreover, their inability to accurately describe complex body positions and movements severely restricts their practical utility in real-world applications requiring detailed human motion understanding. Figure 1. Overview of our proposed M-ACM (Motion-Augmented Caption Model) framework. The system processes input videos through dual pathways: a standard visual pathway (top) and a motion-specialized pathway (bottom). The visual pathway extracts general visual features via a frozen vision encoder, while the motion pathway uses ViTPose-based ( Xu et al. 2022 ) frame sampling and human mesh recovery to generate precise motion representations. Both representations are projected into a common embedding space through trainable projection alignment modules. Our key innovation, Motion Synergetic Decoding (MSD), addresses hallucination issues by comparing logit distributions from both pathways. As shown in the example, without MSD the model incorrectly identifies the basketball being handled with the "foot" (hallucination), whereas with MSD the model correctly identifies the "hand" as the body part manipulating the ball. The dataset limitations further compound these problems, as motion domain datasets are predominantly synthetic collections ( Guo et al. 2022 ; Plappert et al. 2016 ; Mahmood et al. 2019 ; Delmas, Ginger and Weinzaepfel, Philippe and Lucas, Thomas and Moreno-Noguer, Francesc and Rogez, Grégory 2022 ; Delmas, Ginger and Weinzaepfel, Philippe and Moreno-Noguer, Francesc and Rogez, Grégory 2023 ) , which diverge significantly from real-world videos, lacking complex background information and interactions between humans and environments. Existing open-source video datasets ( Chen et al. 2024b ; Grauman et al. 2024 ; Hong* et al. 2024 ; Zhang et al. 2024b ; Ju et al. 2024 ; Chen et al. 2024a ; Chen et al. 2024c ) mostly contain general descriptions without specific annotations for fine-grained human action understanding. Current approaches to mitigating hallucinations ( Chuang et al. 2024 ; He et al. 2024a ; Deng et al. 2025 ; Wang et al. 2025b ; Yin et al. 2023 ) , such as OPERA ( Huang et al. 2024 ) and VCD ( Leng et al. 2024 ) , primarily focus on computationally expensive multiple generation/query iterations without incorporating additional prior information, ultimately failing to address the fundamental issue of inadequate motion representation. In summary, our main contributions are as follows: •Motion Synergetic Decoding (MSD) Framework: A novel approach that reduces hallucination in human motion captioning by combining standard visual understanding with specialized motion representations from SMPL-based human mesh recovery. The dual-pathway system with synergy calculation mechanism significantly improves motion description accuracy. •Human Motion Insight (HMI) Dataset: A comprehensive resource with 115K videos and descriptions focused on human movement, sourced from five diverse datasets. Created through rigorous filtering, motion quality assessment, and ViTPose-based dynamic sampling to facilitate human motion understanding. •HMI-Bench: A specialized benchmark for evaluating motion caption generation across multiple dimensions including detailed movement descriptions, spatial relationships, temporal dynamics, and semantic accuracy. Experiments show the framework outperforms existing methods on both standard and specialized metrics. 2. Related Work 2.1. Video Understanding with Large Language Models Video understanding has emerged as a pivotal research domain in contemporary academia, as it aligns more closely with human perceptual mechanisms for processing real-world information. With the rapid development of multimodal large models, an increasing number of works ( Liu et al. 2024 ; Chen et al. 2024d ; Li et al. 2023 ; Li et al. 2024b ; Li et al. 2024a ; Wang et al. 2024 ; Bai et al. 2023 ; Maaz et al. 2024 ; Zhang et al. 2023 ; Lin et al. 2024 ) have demonstrated remarkable capabilities in video understanding. Early works ( Maaz et al. 2024 ; Zhang et al. 2023 ; Li et al. 2023 ) connect LLM and pretrained vision encoders like CLIP ( Radford et al. 2021 ) through a projection module and finetune the projection module with video instruct data. Towards advancing the development of multimodal large models, VILA ( Lin et al. 2024 ) conducts systematic research