LRC-DHVC: Towards Local Rate Control in Neural Video Compression
Marc Windsheimer, Simon Deniffel, and André Kaup · 2026
Read LRC-DHVC: Towards Local Rate Control in Neural Video Compression by Marc Windsheimer, Simon Deniffel, and André Kaup (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14240.txt.
Introduction
Marc Windsheimer, Simon Deniffel, and André Kaup bring local rate control to neural video compression in 2026. Local rate control generalizes compression for challenges like video coding for machines. Traditional hybrid coding adapts the local rate-distortion trade-off via a local quantization parameter. No such approach is currently available for learning-based video compression.
Their LRC-DHVC aims toward local rate control in neural codecs. Regions that matter could get more bits, and dull regions fewer. Machine viewers need detail in task areas, not in backgrounds. Local control could cut bandwidth for camera networks and analytics.
Catalog record
| CREATOR | Marc Windsheimer, Simon Deniffel, and André Kaup |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-14240
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LRC-DHVC: Towards Local Rate Control in Neural Video Compression Thanks: The authors gratefully acknowledge that this work has been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under project number 426084215. Thanks: The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). The hardware is funded by the German Research Foundation (DFG). Marc Windsheimer, Simon Deniffel, and André Kaup Abstract Local rate control is a key enabler to generalize image and video compression for dedicated challenges, such as video coding for machines. While traditional hybrid video coding can easily adapt the local rate-distortion trade-off by changing the local quantization parameter, no such approach is currently available for learning-based video compression. In this paper, we propose LRC-DHVC, a hierarchical video compression network, which allows continuous local rate control on a pixel level to vary the spatial quality distribution within individual video frames. This is achieved by concatenating a quality map to the input frame and applying a weighted MSE loss which matches the pixelwise trade-off factors in the quality map. During training, the model sees a variety of quality maps due to a constrained-random generation. Our model is the first neural video compression network, which can continuously and spatially adapt to varying quality constraints. Due to the wide quality and bit rate range, a single set of network parameters is sufficient. Compared to single rate point networks, which scale linearly with the number of rate points, the memory requirements for our network parameters remain constant. The code and model are available at link-updated-upon-acceptance . I Introduction Visual data, in particular video content, accounts to the vast majority of all internet traffic [ 1 ] , and as resolutions and frame rates continue to rise, efficient compression becomes ever more critical. Traditional video codecs rely on block-based transforms and handcrafted rate-distortion optimization, but struggle to improve compression performance without adding a significant amount of complexity. With [ 2 ] , neural video compression (NVC) has emerged as alternative, by learning compact latent representations in a end-to-end manner and jointly optimizing transformation, temporal modeling, and entropy coding from training data. However, most existing NVC schemes are either limited to single rate points or manage rate allocation solely using global Lagrangian multipliers across entire sequences or on a frame-wise level. In contrast, local rate control aims to distribute the rate efficiently across both, space and time, tailoring the compression to satisfy scenario-specific quality and rate constraints. In conventional codecs, local rate control is performed using spatial QP adaptation and macroblock-level bit allocation. Translating this ideas to NVC, however, poses unique challenges, since non-stationary latent statistics have to be covered within a single end-to-end compression framework. Local rate control in neural video compression allows wide adoption for many use cases, e.g. for privacy-preserving compression or video coding for machines. II Related Work Fig. 1 : Architectural overview of LRC-DHVC, our hierarchical video codec for local rate control using a pixel-granular quality map \bm{m}_{t} . The latent block incorporates temporal modeling via the temporal priors \bm{F}^{l}_{<t} , probability modeling, quantization and entropy coding. Novel components are highlighted in red. Rate control can be performed on multiple levels in the encoding process of video sequences. Firstly, on the global level for the whole video sequence, secondly, on the frame level and finally, locally for the coding blocks within an individual frame. Hybrid video codecs, such as HEVC [ 3 ] and VVC [ 4 ] , can perform rate control on all of these levels [ 5 ] . Due to the advances in neural image and video compression, rate control methods for neural compression networks are actively developed. So far, rate control algorithms for neural compression mainly focuses on global and frame level optimization. Some variable rate methods for image compression train adaptive compression networks and are controlled by a global quality parameter during inference [ 6 , 7 ] . Other approaches are controlled by specifying a target bit rate to control the rate allocation [ 8 , 9 ] . In [ 10 ] the ideal Lagrangian \lambda_{i} for the i-th frame is derived by modeling the interdependence of rate, distortion, and the Lagrangian. DCVC-FM [ 11 ] applies learned scaling factors in the encoder and decoder transformation, where the scaling factors are indexable for each individual frame. In [ 12 ] , per-frame Lagrangians are derived by matching predicted R - \lambda and R - D relations to a desired target bit rate. Other methods, like code editing [ 13 ] or semi-amortized variational inference [ 14 ] , optimize bit allocation with respect to the traditional global rate-distortion trade-off during inference. While the granularity of rate control in conventional transform coding is limited by the transform block size, neural compression enables rate control for individual pixels. Prior work uses the Spatial Feature Transform (SFT) [ 15 ] , for local rate control [ 16 ] in neural image compression. By training the compression network on a quality map-dependent loss, it can learn to distribute bit rate and quality accordingly. Since the quality map is only an input to the encoder, the network inherently learns to include the required quality information as part of the latent \bm{\hat{y}} and hyperlatent \bm{\hat{z}} . Latent space masking [ 17 ] is used in image coding for machines to shift the latent values towards their predicted mean. While this shift leads to higher distortions, the likelihood of the latent symbols is increased, resulting in a lower rate. Similarly, saliency-dependent compression on varying hierarchical levels [ 18 ] allows a blockwise variation of the rate-distortion trade-off. In [ 19 ] , a Delta-Gaussian entropy model is used in conjunction with a binary region-learning-loss. In neural video compression, existing spatial rate control methods are limited with respect to the adjustment options. ROI-DVC [ 20 ] performs a Region-of-Interest (ROI)-based coding using a binary ROI map of the video frame. The frame and the ROI map are split into multiple patches. For each individual patch, the ratio of ROI area to total patch area is derived and depending on the ratio, one out of three models is selected for encoding the patch. As the different models have been trained with varying rate-distortion trade-offs, patches with a high ROI area are encoded with a higher quality and vice versa. Fig. 2 : Architectural overview of the latent block, where modified components are highlighted in red. MR-MD-SSF [ 21 ] combines a global variable-rate approach with a pixel-accurate binary ROI map, where the relevance of the distortion loss for non-ROI pixels is decreased. This work builds upon the Scale Space Flow model [ 22 ] where the ROI mask is concatenated to the inputs of the I-frame encoder, the flow encoder and the residual coder. Using this input, the model learns to perform adaptive spatial rate allocation. Both designs are limited in terms of spatial rate allocation. For ROI-DVC, the rate-allocation is constrained to the fixed spatial size of the segmented patches, and the rate point options for a specific patch are limited by the number of trained models. While MR-MD-SSF allows higher accuracy in the