Pioneering Perceptual Video Fluency Assessment: A Novel Task with Benchmark Dataset and Baseline

Qizhi Xie, Kun Yuan, Yunpeng Qu, Ming Sun, Chao Zhou, Jihong Zhu · 2026

Read Pioneering Perceptual Video Fluency Assessment: A Novel Task with Benchmark Dataset and Baseline by Qizhi Xie, Kun Yuan, Yunpeng Qu, Ming Sun, Chao Zhou, Jihong Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26055.txt.

Introduction

Xie and co-authors define video fluency assessment as its own task, apart from general video quality. Fluency covers motion smoothness and frame continuity as viewers perceive them. Past quality models fold fluency into one score and understate it, their pilot tests show. They build FluVid, a set of 4,606 wild videos with balanced fluency levels and human scores. They benchmark 23 methods on it. Their baseline, FluNet, uses a temporal permuted self-attention block. It reaches state-of-the-art results, the authors report.

Catalog record

CREATORQizhi Xie, Kun Yuan, Yunpeng Qu, Ming Sun, Chao Zhou, Jihong Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50421
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26055

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pioneering Perceptual Video Fluency Assessment: A Novel Task with Benchmark Dataset and Baseline Qizhi Xie, Kun Yuan, Yunpeng Qu, Ming Sun, Chao Zhou, Jihong Zhu Abstract Accurately estimating humans’ subjective feedback on video fluency , e.g . , motion consistency and frame continuity, is crucial for various applications like streaming and gaming. Yet, it has long been overlooked, as prior arts have focused on solving it in the video quality assessment (VQA) task, merely as a sub-dimension of overall quality. In this work, we conduct pilot experiments and reveal that current VQA predictions largely underrepresent fluency, thereby limiting their applicability. To this end, we pioneer Video Fluency Assessment (VFA) as a standalone perceptual task focused on the temporal dimension. To advance VFA research, 1) we construct a fluency-oriented dataset, FluVid , comprising 4,606 in-the-wild videos with balanced fluency distribution, featuring the first-ever scoring criteria and human study for VFA. 2) We develop a large-scale benchmark of 23 methods, the most comprehensive one thus far on FluVid, gathering insights for VFA-tailored model designs. 3) We propose a baseline model called FluNet , which deploys temporal permuted self-attention (T-PSA) to enrich input fluency information and enhance long-range inter-frame interactions. Our work not only achieves state-of-the-art performance but, more importantly, offers the community a roadmap to explore solutions for VFA. Both FluVid and FluNet will be available at https://github.com/KeiChiTse/VFA . 1 Introduction Video, as a representative multimedia medium, has experienced rapid growth in recent years, playing a pivotal role across various domains, such as social media 4 , content creation 23 , and live e-commerce 54 . According to Cisco’s Visual Networking Index (VNI) 3 , global IP video traffic is expected to account for more than 82% of all IP traffic by 2022. Meanwhile, as video-capturing and viewing devices advance, users increasingly prefer videos that appear to be of “high quality” 9 . The aforementioned trend poses a significant challenge for video providers in improving video quality to enhance users’ Quality of Experience (QoE) 6 . To tackle this challenge, the prevailing approach is to utilize off-the-shelf video quality assessment (VQA) models to quantify quality scores, thereby automatically emulating humans’ subjective feedback on video quality 26 ; 53 ; 76 ; 79 . As VQA research progresses 1 ; 67 ; 85 , numerous factors are identified to affect human-perceived video quality, which can be categorized into frame-wise spatial ( e.g . , noise, color, content composition etc . ) and inter-frame temporal distortions ( e.g . , screen shake, motion inconsistency etc . ). The coexistence of these factors renders VQA models’ quantified scores as a holistic assessment of both spatial and temporal quality, without explicitly disentangling them 46 ; 65 ; 17 . Inspired by prior studies 48 ; 39 ; 50 suggesting that human eyes are more sensitive to temporal than spatial distortion when viewing a video, in this work, through a pioneering benchmark experiment (as in Tab. 4 ), we discover that the entangled prediction of VQA models excels at identifying spatial rather than temporal quality (as in Fig. 1 ). This prohibits the predicted scores from guiding temporal-related downstream tasks in video processing workflows, such as adaptive frame rate encoding 51 and frame interpolation 40 . Figure 1 : Motivation of video fluency assessment: The prevailing VQA paradigm is highly sensitive to spatial distortions, whereas VFA concentrates exclusively on underrepresented temporal factors ( e.g . , camera motion, playback stuttering). To address these limitations of VQA experts, we refer to perceived temporal quality as fluency , and propose a novel perceptual task , named Video Fluency Assessment (VFA) , to emulate the human visual system’s (HVS) perception of fluency accurately. The task can be formulated as follows, where V and y denote the perceived video and its subjective fluency score, respectively: \begin{split}f:V\to y\\ \end{split} (1) The objective of VFA is to learn the mapping function f in Equ. 1 . More importantly, our VFA brings distinctive benefits . First , it can guide video quality enhancement more efficiently. As in Fig. 1 , VFA prediction indicates subsequent efforts should focus on optimizing frame-wise clarity ( e.g . , using denoising operators) rather than fluency ( e.g . , conducting transcoding). Second , VFA prediction can enhance text/image-to-video generation by serving as a fluency assessment metric or a reward for reinforcement learning. In this paper, we dissect the proposed task and pinpoint three problems to advance VFA research: First , there is a lack of a human subjective rating standard for video fluency. As proved in VQA literature, a robust scoring standard—such as ITU-R BT.500 43 —serves as the cornerstone for defining subjective fluency levels and comparing the performance of different assessors 18 ; 50 . Second , the lack of large-scale human studies and datasets with ground-truth fluency scores y presents a challenge, which not only hinders the benchmarking of various approaches but also constrains the full potential of data-driven methods empowered by deep learning 86 ; 70 . Last , current VQA approaches lack explicit model designs to capture fluency, making them inadequate as baselines for the VFA task. To address the 1st problem , we formulate a five-tier accurate category rating (ACR) criteria to mapping human fluency perception ( e.g . , “good”) to numerical scores ( e.g . , 4.0), and vice versa. The definition of five fluency levels is written by a group of 20 visual experts, strictly following the instructions in the ITU regulations 44 ; 45 for correctness. To address the 2nd problem , we conduct the first-ever human study for the VFA task using the proposed criteria, thereby constructing a dataset termed FluVid for performance benchmarking. FluVid comprises over 4,000 videos exhibiting diverse levels of fluency, each accompanied by a Mean Opinion Score (MOS) annotated by 20 experts. Moreover, to mitigate video selection bias and address the long-tailed fluency distribution for robust benchmarking, we meticulously devise a set of principles to guide both video collection and annotation. To address the 3rd problem , through a careful analysis of the performance of 23 models on FluVid, we ascertain that limited input frames and insufficient inter-frame interactions are the obstacles preventing them from perceiving fluency. Therefore, we propose a simple yet powerful baseline model named FluNet . FluNet overcomes the obstacles by a computationally efficient attention mechanism to compress and permute the channels of \mathbf{K} , \mathbf{V} matrices. Moreover, to address the reliance on expert-annotated fluency scores and leave FluVid for benchmarking purposes, we devise a self-supervised (SSL) training strategy . By synthesizing and learning to rank videos with various fluency levels, FluNet surpasses previous methods in VFA by a notable margin in a data-efficient manner. Our main contributions are four-fold: • We pioneer an investigation into the task paradigm of VQA, revealing that current VQA models fall short in quantifying temporal quality. To address this, we formulate temporal quality as fluency and present a novel perceptual task —Video Fluency Assessment (VFA). • To the best of our knowledge, we are the first to establish the subjective fluency scoring standard and conduct a large-scale human study to construct the dataset FluVid , covering over 4,000 videos and fluency-centred MOSs for performance benchmarking.

Read the complete text