AV1 Motion Vector Fidelity and Application for Efficient Optical Flow
Julien Zouein, Vibhoothi Vibhoothi, Anil Kokaram · 2025
Read AV1 Motion Vector Fidelity and Application for Efficient Optical Flow by Julien Zouein, Vibhoothi Vibhoothi, Anil Kokaram (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17427.txt.
Introduction
Compressed video already contains motion estimates, and Julien Zouein, Vibhoothi Vibhoothi, and Anil Kokaram ask how far those estimates can be trusted. Their subject is the AV1 video codec, widely used for streaming, compared against true optical flow, the dense motion field many vision programs need. Reusing the codec numbers could save a great deal of computing time if they prove faithful enough.
Zouein and his co-authors first measure AV1 and the older HEVC codec against known flow, and they show how encoder settings change the faithfulness. They then feed AV1 motion into a leading flow method as a head start. The warm start reached similar accuracy about four times faster, with only a small loss in precision.
Catalog record
| CREATOR | Julien Zouein, Vibhoothi Vibhoothi, Anil Kokaram |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17427
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AV1 Motion Vector Fidelity and Application for Efficient Optical Flow Thanks: This work was funded by the Horizon CL4 2022, EU Project Emerald, 101119800. Julien Zouein, Vibhoothi Vibhoothi, Anil Kokaram Abstract This paper presents a comprehensive analysis of motion vectors extracted from AV1-encoded video streams and their application in accelerating optical flow estimation. We demonstrate that motion vectors from AV1 video codec can serve as a high-quality and computationally efficient substitute for traditional optical flow, a critical but often resource-intensive component in many computer vision pipelines. Our primary contributions are twofold. First, we provide a detailed comparison of motion vectors from both AV1 and HEVC against ground-truth optical flow, establishing their fidelity. In particular we show the impact of encoder settings on motion estimation fidelity and make recommendations about the optimal settings. Second, we show that using these extracted AV1 motion vectors as a “warm-start” for a state-of-the-art deep learning-based optical flow method, RAFT, significantly reduces the time to convergence while achieving comparable accuracy. Specifically, we observe a four-fold speedup in computation time with only a minor trade-off in end-point error. These findings underscore the potential of reusing motion vectors from compressed video as a practical and efficient method for a wide range of motion-aware computer vision applications. I Introduction The estimation of motion between video frames is a fundamental task that bridges the fields of video compression and computer vision. In video compression, motion estimation is essential for exploiting temporal redundancy. In computer vision, the equivalent concept, optical flow, provides vital information for tasks ranging from object tracking to autonomous navigation. However, the computational cost of generating accurate optical flow often creates a significant bottleneck in real-world applications. Given that the vast majority of digital video is stored and transmitted in a compressed format, the motion vectors required for decoding are readily available within the bitstream. This has led to research into reusing these pre-computed motion vectors as a computationally inexpensive proxy for optical flow [ 1 ] . Building on prior work that has primarily focused on older codecs like H.265, this paper investigates the potential of the modern, open-source AV1 codec for this purpose. The main contributions of this paper are: i) A quantitative analysis of the motion vectors extracted from AV1 bitstreams, compared with those from HEVC and ground-truth optical flow, to assess their quality and suitability for computer vision tasks. ii) An investigation into the impact of using these extracted sparse motion fields to accelerate a modern, deep learning-based optical flow algorithm, RAFT, by using the motion vectors as a ”warm-start” to reduce computational overhead. II Background The use of metadata from compressed video bitstreams, particularly motion vectors, for improving the efficiency of computer vision tasks has been an area of active research . The core idea is to leverage the motion information already computed by the video encoder, thus avoiding the computationally expensive process of recalculating it from raw pixel data. In 2014, Kantarov et al. [ 1 ] showed that motion vectors from the H.264 codec could be used for action recognition. By avoiding the analysis of raw pixels, they report an accelerated analysis time with a factor of 2 improvement over traditional feature generation. More recently, Ehrlich et al. have developed a deep learning architecture for compressed video quality enhancement [ 2 ] that leverages structural and motion information embedded in the bitstream to restore fine details in compressed videos. While the use of modern motion estimation techniques in video compression has yielded mixed results [ 3 ] , the idea of exploiting motion information in the bitstream to accelerate optical flow estimation dates back to about 2018 [ 4 ] . Rüfenacht and Taubman proposed HEVC-EPIC [ 4 ] , where they re-used EPICFlow [ 5 ] framework replacing the feature matching part with Motion Vectors extracted from HEVC bitstream, to yield a fast optical flow estimate. In 2020 Taubman refined this approach [ 6 ] , where they reformulate a slow optimization problem into a fast confidence-weighted filtering problem. The use of Deep Learning techniques to refine Motion Vectors extracted from compressed bitstream has also been explored. The RAFT [ 7 ] technique aims to estimate optical flow with high-quality control using an iterative process. In 2022, Zhou et al. [ 8 ] have introduced a framework that uses motion vectors from the bitstream as input to a RAFT-like deep neural network [ 7 ] , which iteratively refines the motion field to produce accurate optical flow. Their system is computationally heavier, using a combination of 3 Convolutional Neural Networks to pre-process motion information before attempting to warm-start their model. Despite the progress in this area, previous work has not investigated the impact of encoder presets and configuration on the quality of the initial sparse motion field using HEVC and AV1. This has a direct impact on the fidelity and usefulness of their motion vectors. Our work addresses these gaps by systematically evaluating the quality of motion vectors produced by modern codecs and by carefully choosing the encoder parameterisation for high-quality, computationally efficient optical flow estimation. III Extraction and pre-processing of the Motion Vectors Hybrid video codecs like AV1 and HEVC share a block-based prediction architecture. AV1 supports a wider range of block sizes (4×4 to 128×128) compared to HEVC (4×4 to 64×64). Each block in a frame can be associated with a motion compensation vector indicating its source location from an already encoded frame (Reference Frame). AV1 supports a total of seven reference frames [ 9 , 10 ] , which can be set up based on coding configuration to be forward or backwards. The key observation is that the raw motion from the bitstream is sparse and unevenly sampled. We therefore need to create a dense, evenly sampled field for further refinement. III-A Motion Vector Extraction and Optic Flow Initialisation We first manipulate the raw (and sparse) motion vectors extracted from the bitstream to create an initial flow field at the pixel resolution. That flow field must contain vectors that reference the previous frame only. Reference Frame Normalisation. Motion vectors (MV) within a single frame n can point to different reference frames (e.g., the previous frame n-1 , or one or several frames in the past n-k ). To create a coherent motion field relative to the immediately preceding frame n-1 , we need to normalise each vector. Given the motion vector v_{n,m}(p,q) at a block (p,q) in frame n , we determine reference frame m by accessing the order_hint syntax element metadata in the bitstream. This maps the reference frame type (e.g., LAST1, LAST2, GOLDEN, ALTREF, etc) to an absolute frame index. Note that order_hint is cyclic, which wraps after 128 frame indexes, this significantly complicates finding m . Given the motion vector v_{n,m}(p,q) we simply determine the normalised vector v_{n,n-1}(p,q) as \displaystyle v_{n,n-1}(p,q)=\frac{v_{n,m}(p,q)}{|n-m|} (1) Handling Missing Motion Data. Blocks encoded using INTRA prediction or designated as SKIPPED do not have associated motion vectors, resulting in (0.0) motion vectors. In some cases the motion in the opposite direction can have non-zero motion. In that case we use Bidirectional Motion Vector Completion (BMVC [ 11 ] ), to infer the missing motion from its corresponding opposite motion vector.