FG-DFPN: Flow Guided Deformable Frame Prediction Network
M. Akın Yılmaz, Ahmet Bilican, A. Murat Tekalp · 2025
Read FG-DFPN: Flow Guided Deformable Frame Prediction Network by M. Akın Yılmaz, Ahmet Bilican, A. Murat Tekalp (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11343.txt.
Introduction
Three researchers present a video prediction method from 2025 that uses motion maps to bend its own filters as it works. Guessing the next frame of a video matters for self-driving systems, video coding, and media tools. Fixed filters struggle when a scene holds many kinds of motion at once, from a panning camera to a small moving object.
Their network first estimates how each pixel moves, then uses that map to steer flexible sampling at several scales. Wide scene shifts and small object moves can thus be handled in one pass. On eight standard test clips, the method beat prior scores by about one unit on a common quality scale. Its speed stayed close to rival methods, the authors report.
Catalog record
| CREATOR | M. Akın Yılmaz, Ahmet Bilican, A. Murat Tekalp |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-11343
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FG-DFPN: Flow Guided Deformable Frame Prediction Network Thanks: A. M. Tekalp acknowledges support from the Turkish Academy of Sciences (TUBA). M. Akın Yılmaz, Ahmet Bilican, A. Murat Tekalp Abstract Video frame prediction remains a fundamental challenge in computer vision with direct implications for autonomous systems, video compression, and media synthesis. We present FG-DFPN, a novel architecture that harnesses the synergy between optical flow estimation and deformable convolutions to model complex spatio-temporal dynamics. By guiding deformable sampling with motion cues, our approach addresses the limitations of fixed-kernel networks when handling diverse motion patterns. The multi-scale design enables FG-DFPN to simultaneously capture global scene transformations and local object movements with remarkable precision. Our experiments demonstrate that FG-DFPN achieves state-of-the-art performance on eight diverse MPEG test sequences, outperforming existing methods by 1dB PSNR while maintaining competitive inference speeds. The integration of motion cues with adaptive geometric transformations makes FG-DFPN a promising solution for next-generation video processing systems that require high-fidelity temporal predictions. The model and instructions to reproduce our results will be released at: https://github.com/KUIS-AI-Tekalp-Research-Group/frame-prediction . I Introduction Video frame prediction aims to forecast future video frames based on one or more past frames. While traditionally formulated as a supervised learning task, it inherently benefits from self-supervision since the ground-truth future frames are available within the video sequence. In complex scenarios—where both fast and slow moving objects, camera motion, and abrupt scene changes coexist—capturing intricate motion patterns becomes particularly challenging. Our proposed Flow-Guided Deformable Frame Prediction Network (FG-DFPN) harnesses the power of optical flow to guide deformable convolution operations. In our design, a dedicated flow estimator module produces coarse motion cues that serve as a reference for learning adaptive offset maps. These offsets dynamically adjust the sampling locations of the convolutional kernels, allowing the network to effectively model non-rigid and multi-scale motion. This flow-guided approach not only stabilizes training but also significantly improves the alignment of multi-scale features, leading to sharper and more accurate future frame predictions. Applications of frame prediction span autonomous driving, where real-time forecasting is critical, to video compression, where precise frame synthesis can reduce the need for transmitting explicit motion information. As demonstrated in Table I , FG-DFPN achieves significantly higher PSNR values compared to all previous video prediction networks, particularly preserving fine details in regions with complex motion and being much faster than all previous methods, which is critical in most contexts. Fig. 1: Our proposed FG-DFPN framework II Related work and Contributions II-A Frame Prediction Video frame prediction has been explored through various paradigms, as summarized in comprehensive reviews [ 1 , 2 ] . Traditional approaches often rely on recurrent convolutional encoder-decoder architectures [ 3 , 4 ] or 3D convolutions [ 5 ] to capture temporal dynamics. Some methods formulate frame prediction as a direct synthesis problem in the pixel domain [ 3 , 4 , 6 , 7 ] , while others explicitly model inter-frame transformations [ 8 , 9 ] . Recent works have further diversified the landscape by leveraging transformation-based and hybrid architectures as well as lightweight designs. Mokssit et al. [ 10 ] propose a spatial transformation-based next frame predictor that disentangles object motion from camera motion by estimating per-object affine and projective transforms—requiring only 14 parameters per object—to robustly predict future frames even under occlusions. Mathai et al. [ 11 ] introduce 3DTransLSTM, a hybrid transformer-LSTM model that employs 3D separable convolutions to efficiently capture long-range spatiotemporal dependencies while significantly reducing model size and computational cost. In a related vein, Mokssit et al. [ 12 ] develop an object-oriented framework that estimates transformation parameters for each detected object, operating in a low-dimensional transformation space rather than directly in the pixel domain, thereby enhancing prediction robustness in dynamic scenes. Complementing these efforts, Mathai et al. [ 13 ] present a lightweight model that integrates separable CNNs with an ST-LSTM architecture to achieve competitive accuracy with a greatly reduced parameter count and complexity. Extending the scope beyond traditional video prediction, Hudson et al. [ 14 ] reformulate a variety of multimodal tasks—spanning text, images, audio, and video—as a unified next-frame prediction problem, enabling a single transformer-based model to seamlessly integrate and process different modalities. Despite progress, many existing approaches still struggle with accurately modeling complex motion, and losses based solely on l_{1} or l_{2} norms [ 4 , 9 ] may lead to blurry predictions. This has spurred research into perceptual and adversarial losses to improve visual fidelity [ 9 , 15 ] . The paper [ 16 ] uses deformable convolutions for modeling geometric transformations, demonstrating that such an approach achieves good performance; we build upon this finding in this paper by enhancing training resilience and improving performance by over 1dB. II-B Flow-Guided Deformable Convolutions Deformable convolution [ 17 ] introduces learnable offsets to the conventional convolutional grid, allowing the network to adapt its receptive field to varying motion patterns. Deformable convolution applied to object detection and video restoration [ 18 , 19 ] , video compression [ 20 ] , and video prediction [ 16 ] . Using flow guidance for the offsets have been demonstrated to achieve much better results in video super-resolution [ 21 ] , and compression [ 22 ] . In FG-DFPN, optical flow is used to guide the deformable convolutions: a Flow Estimator (FlowNET) predicts coarse flow, which is then incorporated into offset prediction for the deformable convolution layers. This strategy ensures that the learned offsets remain consistent with the underlying motion, effectively aligning features across frames and mitigating misalignment artifacts inherent in pure flow-based warping. Fig. 2: Visual comparison of predicted frames for the Garden sequence. Top row shows the prediction results from: (a) DiffCNN (PSNR: 26.48, SSIM: 0.922), (b) DFPN (PSNR: 26.76, SSIM: 0.936), and (c) our proposed FG-DFPN (PSNR: 28.78, SSIM: 0.964). Bottom row (d-f) displays the corresponding error maps between the ground truth and predictions from each method, clearly demonstrating the superior performance of our approach with significantly reduced errors. II-C Contributions Our main contributions are summarized as follows: • We propose the first Flow-Guided Deformable Frame Prediction Network (FG-DFPN) that combines explicit optical flow guidance with adaptive deformable convolutions to capture complex spatiotemporal dynamics. • We introduce a multi-scale architecture that integrates a Flow Estimator, a Multi-Scale Feature Extractor, an Offset and Mask Predictor, and novel Offset Diversity modules. This design allows the network to fuse both global motion trends and fine local details, leading to superior prediction performance. • Extensive experiments demonstrate that FG-DFPN outperforms conventional fully convolutional [ 4 , 7 ] and deformable transformation based [ 16 ] networks in terms of prediction accuracy, runtime ,and computational efficiency, making it a promising solution for applications such as autonomous driving and video compression. III Proposed Method Our video prediction framework processes four consecutive grayscale frames and is built from several key components that work in tandem to capture multi-scale motion and fuse information via deformable convolutions. Figure 1 provides an overview of the architecture. In brief, the system first estimates coarse motion using an optical flow estimator, then extracts multi-scale features from each input frame, warps these features according to the estimated flows, predicts deformable offsets and modulation masks, fuses the aligned features via flow-guided deformable convolutions, and