On Self-Adaptive Perception Loss Function for Sequential Lossy Compression

Sadaf Salehkalaibar, Buu Phan, Likun Cai, Joao Atz Dick, Wei Yu, Jun Chen, Ashish Khisti · 2025

Read On Self-Adaptive Perception Loss Function for Sequential Lossy Compression by Sadaf Salehkalaibar, Buu Phan, Likun Cai, Joao Atz Dick, Wei Yu, Jun Chen, Ashish Khisti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10628.txt.

Introduction

Sadaf Salehkalaibar and co-authors design a smarter realism score for video compression in a 2025 computing paper. Streaming compression must shrink each frame as it arrives, with no peeking ahead. Judging realism by single frames misses how errors stick around across a sequence. Their score judges each new frame jointly with the frames rebuilt before it.

Theory for simple signal models shows the new score avoids frozen errors that older scores allow. It also exploits links between a sharp current frame and its neighbors. Tests on moving digit clips and standard video sets backed the math. Live streams and calls could look cleaner at the same bit budget with such a guide.

Catalog record

CREATORSadaf Salehkalaibar, Buu Phan, Likun Cai, Joao Atz Dick, Wei Yu, Jun Chen, Ashish Khisti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE147 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10628

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Self-Adaptive Perception Loss Function for Sequential Lossy Compression Sadaf Salehkalaibar, Buu Phan, Likun Cai, Joao Atz Dick, Wei Yu, Jun Chen, Ashish Khisti Abstract We consider causal, low-latency, sequential lossy compression, with mean squared-error (MSE) as the distortion loss, and a perception loss function (PLF) to enhance the realism of reconstructions. As the main contribution, we propose and analyze a new PLF that considers the joint distribution between the current source frame and the previous reconstructions. We establish the theoretical rate-distortion-perception function for first-order Markov sources and analyze the Gaussian model in detail. From a qualitative perspective, the proposed metric can simultaneously avoid the error-permanence phenomenon and also better exploit the temporal correlation between high-quality reconstructions. The proposed metric is referred to as self-adaptive perception loss function (PLF-SA), as its behavior adapts to the quality of reconstructed frames. We provide a detailed comparison of the proposed perception loss function with previous approaches through both information theoretic analysis as well as experiments involving moving MNIST and UVG datasets. 1 Introduction In recent years, the topic of lossy compression for videos has received significant attention, driven by the growing demand for producing visually appealing reconstructions even at lower bitrates. Early versions of compression algorithms relied on distortion measures, e.g., mean squared error (MSE), MS-SSIM ( Golinski et al., 2020 ; Rippel et al., 2021 ; Li et al., 2021 ) and PSNR ( Agustsson et al., 2020a ; Yang et al., 2020 ; Rippel et al., 2021 ; Li et al., 2021 ) . However, these metrics often resulted in outputs that were perceived as blurry and lacking realism . Consequently, there have been efforts to incorporate perception -based loss functions into compression systems to improve visual quality. These loss functions aim to quantify the divergence between the distributions of the source and the reconstruction, where achieving perfect perceptual quality means that the two distributions match with each other. Blau and Michaeli ( Blau & Michaeli, 2019 ) explored the rate-distortion-perception (RDP) tradeoff from a theoretical perspective. Subsequently, Zhang et al. ( Zhang et al., 2021 ) introduced universal representations, wherein the representation remains fixed during encoding, and only the decoder can be adjusted to attain near-optimal performance. With the multitude of frames in a video, there is no unique perception loss function (PLF) that is suitable in all cases. In fact, at least two different PLFs have been proposed in prior work. One choice is a PLF metric based on the framewise marginal distributions (FMD) of the source and reconstruction ( Mentzer et al., 2022 ) , where the perception loss function only preserves the marginal distribution of the reconstructed frames instead of the joint distribution. In contrast to this choice, other works ( Veerabadran et al., 2021 ) have proposed PLF based on the entire joint distribution (JD) of source frames. A recent study ( Salehkalaibar et al., 2023 ) establishes the rate-distortion-perception (RDP) trade-off for both metrics. It is shown that at low bitrates, PLF-JD encounters error permanence phenomenon , where errors propagate across all future reconstructions, leaving distortion unchanged across frames. On the other hand, at higher bit rates PLF-JD is more desirable, as PLF-FMD does not address temporal consistency between frames. (a) (b) (c) Figure 1 : (a) Outputs for MovingMNIST with the first frame compressed at a low bitrate R_{1}=12 bits. PLF-SA and PLF-FMD recover from previous errors, while PLF-JD and DCVC-HEM exhibit error permanence. (b) Outputs for UVG with the first frame compressed at a low bitrate R_{1}=0.144 bpp. PLF-SA and PLF-FMD maintain color tone, whereas PLF-JD propagates color tone errors. DCVC-HEM struggles to reconstruct details like eye pupils, while PLF models perform better. (c) Outputs for MovingMNIST with the first frame compressed at a high bitrate R_{1}=\infty bits. PLF-FMD produces reconstruction error without maintaining the temporal correlation. PLF-JD propagates the trajectory error while PLF-SA rectifies the error preserves the temporal correlation across different frames. In this work, we study causal, low-latency, sequential video compression when the output is subjected to both a mean squared-error (MSE) distortion loss and a new perception loss function, which we refer to as Self-Adaptive (SA) . Our proposed metric (PLF-SA) considers the joint distribution between the current source frame and the previous reconstructions. We establish the rate-distortion-perception function for first-order Markov sources and analyze the Gaussian source in detail for our proposed PLF. We also present experimental results involving moving MNIST and UVG datasets. Our key observation is that our proposed PLF mitigates the disadvantages of previously proposed metrics: 1) when the previous reconstructions are of lower quality our proposed PLF does not suffer from the error permanence phenomenon observed with PLF-JD; 2) when the previous reconstructions are of higher quality our proposed PLF preserves the joint distribution with these frames and yields better temporal consistency than PLF-FMD. We summarize these below: • Resilience to Error Permanence Phenomenon : Using theoretical analysis of the rate-distortion-perception function of first-order Gauss-Markov sources and through experiments (see, e.g., Fig. 1 a and Fig. 1 b), we demonstrate that PLF-SA does not suffer from the error permanence phenomenon. In particular, when the first source frame is compressed at a low bitrate, PLF-JD fails to correct mistakes appearing in this frame in subsequent reconstructions. PLF-SA does not suffer from this effect. • Sensitivity to Temporal Correlation across Frames : Through both theoretical analysis and experimental findings (see Fig. 1 c), we demonstrate that our proposed PLF-SA metric can better exploit temporal correlation across frames to yield improved reconstruction. In this setting we assume that the first frame is compressed at a higher bitrate while the second frame is compressed at a lower bitrate. We note that while PLF-FMD yields incorrect output in the second frame, PLF-SA is able to exploit the temporal correlation with the first frame to output the correct digit in the second frame. We also note that PLF-JD still suffers from error permanence as observed in the incorrect trajectory in the reconstruction of the third frame in Fig. 1 c; PLF-SA also does not suffer from this effect. PLF-SA does not suffer from the error permanence phenomenon at low bitrates and maintains temporal correlation among frames, especially when the first frame undergoes high-rate compression. Consequently, it takes advantage of both metrics (PLF-FMD or PLF-JD), depending on the operating rate regime. This adaptability to varying rates is the rationale behind naming this PLF as Self-Adaptive. 2 System Model Assume that we have T frames of video denoted by (X_{1},\ldots,X_{T})\in\mathcal{X}_{1}\times\ldots\times\mathcal{X}_{T} (where \mathcal{X}_{i}\subseteq{\mathbbm{R}}^{d} ) distributed according to joint distribution P_{X_{1}\ldots X_{T}} . The encoders and decoders have access to a shared common randomness K\in\mathcal{K} . The (possibly stochastic) j th encoding function gets the sources (X_{1},\ldots,X_{j}) and the key K and outputs a variable length message M_{j}\in\mathcal{M}_{j}(=\{0,1\}^{\star}) , i.e., \displaystyle f_{j} \displaystyle\colon \displaystyle\mathcal{X}_{1}\times\ldots\times\mathcal{X}_{j}\times\mathcal{K}\to\mathcal{M}_{j},\qquad j=1,\ldots,T. (1) The j th decoding function receives the messages (M_{1},\ldots,M_{j}) and using the key K , it outputs a reconstruction \hat{X}_{j}\in\hat{\mathcal{X}}_{j}(\subseteq{\mathbbm{R}}^{d}) , i.e., \displaystyle g_{j} \displaystyle\colon \displaystyle\mathcal{M}_{1}\times\mathcal{M}_{2}\times\ldots\times\mathcal{M}_{j}\times\mathcal{K}\to\hat{\mathcal{X}}_{j}.\;\; (2) The mappings \{f_{j}\}_{j=1}^{T} and \{g_{j}\}_{j=1}^{T} induce the conditional distribution P_{\hat{X}_{1}\ldots\hat{X}_{T}|X_{1}\ldots X_{T}} for the reconstructed video given the original video. The proposed framework illustrated in Fig. 2 is a one-shot setting i.e., a single sample of the source is compressed at a

Read the complete text