Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training
Shota Hirose, Kazuki Kotoyori, Kasidis Arunruangsirilert, Fangzheng Lin, Heming Sun, Jiro Katto · 2025
Read Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training by Shota Hirose, Kazuki Kotoyori, Kasidis Arunruangsirilert, Fangzheng Lin, Heming Sun, Jiro Katto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23185.txt.
Introduction
Every network delay between a camera and a distant screen taxes the person watching, and predicting the next frames can hide the tax. Shota Hirose and five co-authors build IFRVP in 2025, a real time video predictor grown from frame interpolation models rather than heavyweight forecasters. Three training recipes convert an interpolator into a predictor, and lean residual blocks speed the result.
In their tests the models took the best balance of accuracy and speed among known methods. A demonstration video accompanies the paper. The authors aim at zero perceived latency for remote interaction and control. Prediction, they argue, is cheaper than light speed. The trick is doing it fast enough to matter.
Catalog record
| CREATOR | Shota Hirose, Kazuki Kotoyori, Kasidis Arunruangsirilert, Fangzheng Lin, Heming Sun, Jiro Katto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23185
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Real-time Video Prediction With Fast Video Interpolation Model and Prediction Training Shota Hirose, Kazuki Kotoyori, Kasidis Arunruangsirilert, Fangzheng Lin, Heming Sun, Jiro Katto Abstract Transmission latency significantly affects users’ quality of experience in real-time interaction and actuation. As latency is principally inevitable, video prediction can be utilized to mitigate the latency and ultimately enable zero-latency transmission. However, most of the existing video prediction methods are computationally expensive and impractical for real-time applications. In this work, we therefore propose real-time video prediction towards the zero-latency interaction over networks, called IFRVP (Intermediate Feature Refinement Video Prediction). Firstly, we propose three training methods for video prediction that extend frame interpolation models, where we utilize a simple convolution-only frame interpolation network based on IFRNet. Secondly, we introduce ELAN-based residual blocks into the prediction models to improve both inference speed and accuracy. Our evaluations show that our proposed models perform efficiently and achieve the best trade-off between prediction accuracy and computational speed among the existing video prediction methods. A demonstration movie is also provided at http://bit.ly/IFRVPDemo . The code will be released at https://github.com/FykAikawa/ IFRVP . \bstctlcite IEEEexample:BSTcontrol † † address: 1 School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan 2 School of Engineering, Tokyo Institute of Technology, Tokyo, Japan 3 Yokohama National University, Kanagawa, Japan 1 Introduction Advancement in telecommunication technology drives implementations of video communication in many applications, such as video surveillance, virtual conferences, remote working, online gaming, real-time live streaming, etc. In these applications, reducing latency is not only highly beneficial for real-time interaction, but also especially significant in mission-critical systems such as remote surgery, remote operation with haptic devices, and self-driving vehicles: low-latency telemetry data is required for safety operations. Figure 1 : MS-SSIM on Cityscapes [ 1 ] (resized to 512 \times 1024) vs. computational complexity of various video prediction models. Proposed methods are marked as \mathchoice{\mathbin{\vbox{\hbox{\scalebox{1}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{1}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{1}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{1}{$\scriptscriptstyle\bullet$}}}}} . We use the reported values from [ 2 ] for other methods. Many factors, including network congestion and processing delays, resulted in latency. While advancements in network technology such as the 5G New Radio (NR) network, Beyond 5G (or 6G), and all-optical networks greatly reduce end-to-end latency, zero latency still remains technically impossible due to the physical speed limit of radio waves and light, and although new hardware codecs greatly reduce video processing delay, the increasing demand for higher video quality, e.g., 4K and 8K contents, generally offsets these improvements. Currently, we are attempting to demonstrate zero-latency interactive transmission by combining low-latency network technology with deep-learning-based prediction at the application layer [ 3 , 4 , 5 , 6 ] . Independently, recent work called ZGaming [ 7 ] shows that video prediction is a feasible option to enable zero latency cloud gaming experiences by predicting synthetic foreground objects in the virtual world. We consider that video and haptic prediction are the keys to enabling zero latency interaction among remote locations by compensating for network propagation delays. Early video prediction techniques, such as PredNet [ 8 ] and CrevNet [ 9 ] , introduce state models to predict the unknown future state. However, recently, video prediction models, such as CorrWise [ 10 ] and DMVFN [ 2 ] , have switched to a stateless model. Furthermore, there are approaches that utilize extra information such as semantic segmentation maps in addition to RGB frames [ 11 , 12 ] . In this paper, we propose a simple neural network for video prediction based on the stateless model, called IFRVP, which is an extension of IFRNet [ 13 ] , that can be inferred in real-time. We first propose three model training methods that extend a video interpolation model to handle video prediction tasks and compare them from multiple viewpoints including prediction accuracy, memory (VRAM usage), and computational cost. Then, we show that the recursive model suffers from error accumulation when predicting distant frames, while other models are immunized and highly preferable in zero-latency applications. After that, we introduce an ELAN-based residual block [ 14 ] , which is commonly used in object detection networks, to improve the performance and precision of our prediction network. Our main contributions are listed below: • We propose three new methods to train video frame interpolation models for video prediction as follows: 1. Recurrent Prediction , which uses the two latest frames and recursively predicts the next frame. 2. Arbitrary Prediction , which enables prediction of any timestep in a single inference by timestep embedding and yields faster inference speed and higher accuracy at large timesteps. 3. Independent Prediction , which utilizes multiple models corresponding with possible timesteps. This enables prediction of any timestep in a single inference and achieves the best accuracy of the three methods due to larger model capacity. • We propose an ELAN-based residual block, which is lightweight and efficient, and implemented in the IFRVP model. • We achieved a state-of-the-art speed-accuracy trade-off, offering 20% faster inference speed, while yielding a higher MS-SSIM than DMVFN. 2 Background 2.1 Video Prediction Video prediction has been the main interest of many researchers during the last ten years [ 15 ] . Possible applications include collision avoidance in autonomous driving, which enhances operational safety. As mentioned earlier, early video prediction models utilize the model architecture with internal states like LSTM to exploit spatio-temporal relations, such as PredNet [ 8 ] and CrevNet [ 9 ] . However, this approach is computationally intensive and also requires pre-conditioning frames as the input. To alleviate this problem, simplified and stateless video prediction models have been proposed, such as DVF [ 16 ] , OPT [ 17 ] , CorrWise [ 10 ] , and DMVFN [ 2 ] . Instead of relying on previous states, these model takes advantage of optical flows to handle the prediction tasks. 2.2 Frame Interpolation and Extrapolation Frame interpolation can be classified into optical-flow-based methods and kernel-based methods [ 18 ] . In optical-flow-base methods, the frame interpolation can be formulated as follows. We denote reference RGB frames by I_{0} and I_{1} , and target timestep by t , where 0\leq t\leq 1 . After that, We denote backward warping by \overleftarrow{\mathcal{W}} , optical flows by F_{t\rightarrow 0} and F_{t\rightarrow 1} , and a pixel weight map by M , where 0\leq M\leq 1 . Hence, the interpolated frame \hat{I}_{t} is given by: {\hat{I}}_{t}=M\odot{\hat{I}}_{t\leftarrow 0}+(1-M)\odot{\hat{I}}_{t\leftarrow 1} {\hat{I}}_{t\leftarrow 0}=\overleftarrow{\mathcal{W}}(I_{0},F_{t\rightarrow 0}),\qquad{\hat{I}}_{t\leftarrow 1}=\overleftarrow{\mathcal{W}}(I_{1},F_{t\rightarrow 1}) where M is generally proportional to the distance to the reference frames. Recent works utilize neural networks to improve the prediction accuracy of the optical flow F . RIFE [ 19 ] applies a coarse-to-fine network to predict the optical flow, interpolates pixels, and refines the interpolated pixels by the additional network. IFRNet [ 13 ] extends RIFE by unifying the optical flow estimation and the pixel refinements into a single network, significantly improving the performance. SLAMP [ 20 ] also employs the optical flow for video prediction, but one of the warped images is replaced by the appearance prediction that is effective in static regions. SLAMP also utilizes LSTM to manage temporal probability models, increasing model complexity as mentioned previously. Voxel flow is an extension of the optical flow that represents 3D spatio-temporal relations of pixels, which is also utilized in