CWRNN-INVR: A Coupled WarpRNN based Implicit Neural Video Representation

Yiyang Li, Yanbo Gao, Shuai Li, Zhenyu Du, Jinglin ZhangHui Yuan, , Mao Ye, , Xingyu Gao · 2026

Read CWRNN-INVR: A Coupled WarpRNN based Implicit Neural Video Representation by Yiyang Li, Yanbo Gao, Shuai Li, Zhenyu Du, Jinglin ZhangHui Yuan, , Mao Ye, , Xingyu Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06564.txt.

Introduction

Li and seven colleagues build CWRNN-INVR. Implicit neural video representation uses learnable grids and networks, yet existing methods ignore each part's role. The team first compares network and grid variants: networks capture general structure, grids specific detail. Their mixed framework uses the network for regular information and a residual grid for irregular information. A coupled WarpRNN motion module represents regular structured information, while a mixed residual grid learns irregular appearance and motion together. Experiments show the best reconstruction, averaging 33.73 decibels on UVG under the 3M model.

Catalog record

CREATORYiyang Li, Yanbo Gao, Shuai Li, Zhenyu Du, Jinglin ZhangHui Yuan, , Mao Ye, , Xingyu Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41087
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06564

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CWRNN-INVR: A Coupled WarpRNN based Implicit Neural Video Representation Yiyang Li, Yanbo Gao, Shuai Li, Zhenyu Du, Jinglin ZhangHui Yuan, , Mao Ye, , Xingyu Gao Abstract Implicit Neural Video Representation (INVR) has emerged as a novel approach for video representation and compression, using learnable grids and neural networks. Existing methods focus on developing new grid structures efficient for latent representation and neural network architectures with large representation capability, lacking the study on their roles in video representation. In this paper, the difference between INVR based on neural network and INVR based on grid is first investigated from the perspective of video information composition to specify their own advantages, i.e., neural network for general structure while grid for specific detail. Accordingly, an INVR based on mixed neural network and residual grid framework is proposed, where the neural network is used to represent the regular and structured information and the residual grid is used to represent the remaining irregular information in a video. A Coupled WarpRNN-based multi-scale motion representation and compensation module is specifically designed to explicitly represent the regular and structured information, thus terming our method as CWRNN-INVR. For the irregular information, a mixed residual grid is learned where the irregular appearance and motion information are represented together. The mixed residual grid can be combined with the coupled WarpRNN in a way that allows for network reuse. Experiments show that our method achieves the best reconstruction results compared with the existing methods, with an average PSNR of 33.73 dB on the UVG dataset under the 3M model and outperforms existing INVR methods in other downstream tasks. The code can be found at https://github.com/yiyang-sdu/CWRNN-INVR.git. Fig. 1: Illustration of using different combinations of grids and neural networks (only NeRVBlocks) while keeping the size of the total model constant, for video reconstruction on the Big Buck Bunny dataset. Performance varies significantly with different combinations, indicating that grids and networks each have their own advantages in terms of feature representation. I Introduction Implicit Neural Representations (INRs) [ 1 , 2 , 8 , 4 , 22 , 29 ] have gained increasing interests recently due to its strong ability in learning feature representations. Among the various signal types represented by INR, Implicit Neural Video Representation (INVR) [ 2 , 3 , 6 , 7 , 9 , 15 ] to fit a video has been widely studied. Existing INVR methods [ 17 , 41 , 43 , 64 , 66 ] can be broadly categorized into two approaches considering their representation formats. The first approach takes the frame index as input and uses MLP or other neural networks to reconstruct the video frame at each frame index. This approach focuses on developing efficient neural networks to represent videos, and can be termed as Implicit Neural Video Representations based on Neural Network (INVR-N), as in [ 2 , 7 ] . The other approach uses a set or multiple sets of learnable variables/embeddings, such as grids or codes, as the 2D video frame features, which are then processed by NeRVBlocks to reconstruct frames [ 3 , 15 , 43 ] . Taking HNeRV as an example, it uses an encoder to explicitly produce a latent embedding as input for the representation. This approach focuses on constructing efficient grid representations while only using the neural network as a decoder, and can be referred to as Implicit Neural Video Representation based on Grid (INVR-G) [ 61 ] . However, the differences between using neural networks and grids have not been thoroughly investigated, and the benefits of both representations are not fully explored yet. From the perspective of representation, a video sequence actually contains two types of information. One is the structured information with regular patterns that can be easily represented by a neural network in the spatiotemporal domain, including structured appearance and structured motion that regularly exist across frames [ 11 , 14 , 18 , 20 ] . This kind of information can be well learned by neural networks. The other is irregular information with patterns hard to be represented by a neural network, including unique local appearance and varying motion that only irregularly exist in one or serval frames. Such information cannot be efficiently represented by a neural network since they only occur occasionally in a video. In such a case, a specifically learned grid can be used to represent the irregular information with a small representation cost. Fig. 2: Illustration of the spatial and temporal regular and irregular information in a video. The top shows the comparison of the reconstruction results using neural network only (without grid) and with grid (under the same number of parameters in total) on the UVG Beauty video. The residual map is obtained by subtracting the two results. The bottom is the reconstruction results on a spliced video that n frames of the Beauty are inserted between the 2N frames of a blooming lily video. The residual map in the top illustrates the spatial irregular information, mainly on the hair which is hard to be represented and not structured among video frames. The bottom results using the spliced video clearly show the temporal irregular information, which is difficult to be represented by a neural network but easily represented by a grid. Fig. 1 illustrates using different combinations of grids and neural networks for video reconstruction on the Big Buck Bunny dataset [ 26 ] where the combination is measured by the percentage of grid parameters and network parameters (grid/network ratios). A smaller network leads to a large ratio and vice versa. It can be seen that performance varies significantly with different combinations, indicating that grids and networks each have their own advantages in terms of feature representation. By comparing a pair of large and small grid/network ratios, it can be seen that the performance dropped significantly, i.e., using a smaller neural network is much worse than using a smaller grid at a similar ratio, 32.02 dB versus 30.53 dB under ratios of 20% and 80%, respectively. This illustrates that neural network is more efficient in representing regular and structured information than grid since the overall quality is largely controlled by the regular information repeatedly occurred in a video. To further demonstrate the structured information with regular patterns and irregular information, Fig. 2 illustrates the reconstruction results using neural network only (without grid) and with grid (under the same number of parameters in total). By comparing the face (easy to represent and regularly occurred) and hair (difficult to represent and changed over frames) in the Beauty video, it can be seen that the model with Grid can better capture more spatial irregular information details than using neural network only. The residual map is further demonstrated in the right to show the differences. To better visualize the temporal irregular information, a video created by inserting n frames of the Beauty video between 2N frames of a blooming lily video is used. The results using neural network only (without grid) and with Grid are illustrated. It can be clearly seen that when only one frame is inserted ( n=1 ), the frame is constructed mostly with the temporal regular information, which is the Lily flower in the video. Even when n=10 ,

Read the complete text