Per-channel autoregressive linear prediction padding in tiled CNN processing of 2D spatial data

Olli Niemitalo, Otto Rosenberg, Nathaniel Narra, Olli Koskela, Iivari Kunttu · 2025

Read Per-channel autoregressive linear prediction padding in tiled CNN processing of 2D spatial data by Olli Niemitalo, Otto Rosenberg, Nathaniel Narra, Olli Koskela, Iivari Kunttu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12300.txt.

Introduction

Olli Niemitalo and co-authors pad image tiles by prediction in a 2025 paper. Neural image processors work tile by tile, and each tile's edge needs padding that guesses what lies beyond. Flat or copied edges guess poorly and waste model effort.

Their padding fits a small prediction model per color channel to the known edge pixels, then fills the border with its forecasts. A satellite photo upscaler trained with the method edged out zero and copy padding on error, at moderate extra time. Cropping a few output pixels beat all padding tricks for this load, though. The paper maps when prediction padding pays.

Catalog record

CREATOROlli Niemitalo, Otto Rosenberg, Nathaniel Narra, Olli Koskela, Iivari Kunttu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12300

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Per-channel autoregressive linear prediction padding in tiled CNN processing of 2D spatial data Olli Niemitalo, Otto Rosenberg, Nathaniel Narra, Olli Koskela, Iivari Kunttu Abstract We present linear prediction as a differentiable padding method. For each channel, a stochastic autoregressive linear model is fitted to the padding input by minimizing its noise terms in the least-squares sense. The padding is formed from the expected values of the autoregressive model given the known pixels. We trained the convolutional RVSR super-resolution model from scratch on satellite image data, using different padding methods. Linear prediction padding slightly reduced the mean square super-resolution error compared to zero and replication padding, with a moderate increase in time cost. Linear prediction padding better approximated satellite image data and RVSR feature map data. With zero padding, RVSR appeared to use more of its capacity to compensate for the high approximation error. Cropping the network output by a few pixels reduced the super-resolution error and the effect of the choice of padding method on the error, favoring output cropping with the faster replication and zero padding methods, for the studied workload. 1 Introduction Figure 1: Satellite images padded using our linear prediction padding method (variant lp6x7 ). Geospatial rasters and other extensive spatial data can be processed in tiles (patches) to work around memory limitations. The results are seamless if all whole-pixel shifts of the tiling grid result in the same stitched results. A convolutional neural network (CNN) consisting of valid convolutions and pointwise operations is equivariant to whole-pixel shifts. In such shift equivariant CNN-based processing, each input tile must cover the receptive fields of the output pixels, i.e. input tiles must be overlapped. This wastefully repeats computations. In deep CNNs, the receptive fields may be thousands of pixels wide ( Araujo et al. 2019 ), exacerbating the problem. Input Output Valid convo- lution Figure 2: Valid convolution with a 3{\times 3} kernel with origin in the middle erodes data spatially by a one-pixel-thick layer at each image edge. The receptive field of a single output pixel ( ) is illustrated. Spatial reduction (see Fig. 2 ) in deep CNNs is commonly compensated for by padding the input of each spatial convolution, with zeros in the case of the typically used zero padding ( zero for brevity) or with the value of the nearest input pixel in replication padding ( repl ). In the less used polynomial extrapolation padding ( extr N ), a Lagrange polynomial of a degree N-1 is fitted to the N nearest input pixels from the same row or column, and the padding is sampled from the extrapolated polynomial. extr0 is equivalent to zero and extr1 to repl . An ideal padding method would exactly predict the spatial data outside the padding input view. This is not possible to do for data coming from an effectively random process, such as natural data or a CNN feature map derived from it. Using padding increases processing error towards tile edges ( Huang et al. 2018 ) and will generally break CNN shift equivariance. Center-cropping the CNN output reduces the error ( Huang et al. 2018 ) and thus indirectly improves shift equivariance. The strength of the receptive fields of output neurons typically decay super-exponentially with distance to their center ( Luo et al. 2016 ), meaning that an output center crop less than the radius of the theoretical receptive field could reduce the processing error close to that of a valid-convolution CNN. In this work we introduce linear prediction padding (abbreviated lp , see Fig. 1 ) and evaluate the performance of tiled CNN super-resolution employing padding methods lp , zero , repl , and extr . 2 Linear prediction padding Linear prediction is a method for recursively predicting data using nearby known or already predicted data as input (for an introduction, see Makhoul 1975 for the 1D case and Weinlich 2022 for the 2D case, both very approachable texts). Linear prediction is closely related to stochastic autoregressive (AR) processes. In the 2D case, we can model single-channel image or feature map data I , assumed real-valued, by a zero-mean stationary process \hat{I} : \hat{I}_{(y,x)+h_{0}}=\sum_{i=1}^{P}a_{i}\hat{I}_{(y,x)+h_{i}}+\varepsilon_{(y,x)+h_{0}}, (1) where (y,x) are integer spatial coordinates, a_{i} are coefficients that parameterize the process, \varepsilon is zero-mean independent and identically distributed (IID) noise, and the extended neighborhood h is a list of 2D coordinate offsets (relative to a shared origin), first the pixel of interest \hat{I}_{(y,x)+h_{0}} , followed by its neighborhood \hat{I}_{(y,x)+h_{1\ldots P}} in any order. Each pixel depends linearly on P neighbors (see Fig. 3 ). Our approach to linear prediction padding is to least-squares (LS) fit the AR model (Eq. 1 ) to actual image data I and to compute the padding as conditional expectancies assuming that the data obeys the model, \hat{I}=I . 1{\times}1 2{\times}1 2{\times}3 2{\times}5 3{\times}3 Figure 3: Illustration of some rectangular 2D neighborhoods (with pixels ) next to the pixel of interest ( ), defining the linear dependency structure of a downwards causal AR model. The extended neighborhood 1{\times}1 can be defined by h=[(1,0),(0,0)] and 2{\times}1 by h=[(2,0),(0,0),(1,0)] . The LS fit is obtained by minimizing the mean square prediction error (MSE), with residual noise as the error: \begin{gathered}\varepsilon_{(y,x)+h_{0}}=\sum_{i=1}^{P}a_{i}I_{(y,x)+h_{i}}-I_{(y,x)+h_{0}},\quad\text{MSE}={\textstyle\frac{1}{|S|}}\hskip-7.5pt\sum_{(y,x)\in S}\hskip-7.5pt\Big(\sum_{i=1}^{P}a_{i}I_{(y,x)+h_{i}}-I_{(y,x)+h_{0}}\Big)^{2},\\ S=\left\{(y,x)\,|\,(y,x)+h_{i}\in K\text{ for all }i\in[0,P]\right\}\end{gathered} (2) where S is the set of coordinates that keeps all pixel accesses in the sums within the set of input image pixel coordinates K (see the left side of Fig. 4 ). The noise, being zero-mean by definition, does not contribute to the AR process expected value: \begin{split}\operatorname{E}\big(\hat{I}_{(y,x)+h_{0}}\big)=&\operatorname{E}\Bigg(\sum_{i=1}^{P}a_{i}\hat{I}_{(y,x)+h_{i}}+\varepsilon_{(y,x)+h_{0}}\Bigg)=\sum_{i=1}^{P}a_{i}\operatorname{E}\big(\hat{I}_{(y,x)+h_{i}}\big)+0.\end{split} (3) We can recursively calculate conditional expectances (the padding) further away from K using both the known pixels (the input image) and any already calculated expectances (the nascent padding): \begin{gathered}\operatorname{E}\big(\hat{I}_{(y,x)+h_{0}}\,\big|\,\hat{I}_{(y,x)\in K}\big)=\sum_{i=1}^{P}a_{i}\begin{cases}\hat{I}_{(y,x)+h_{i}}&\text{if\,}{\scriptstyle(y,x)+h_{i}}\in K,\\ \operatorname{E}\big(\hat{I}_{(y,x)+h_{i}}\,\big|\,\hat{I}_{(y,x)\in K}\big)&\text{otherwise}.\end{cases}\end{gathered} (4) \ldots \ldots Figure 4: Downwards linear prediction padding with a 4{\times}5 neighborhood — Left: predicted ( ) and neighborhood pixels ( ) at the corners of the rectangular area of coordinates over which MSE is calculated during fitting. Right: corner handling prevents narrowing of the recursive prediction front. We use rotated extended neighborhoods to pad in different directions, and adjust the extended neighborhoods (see the right side of Fig. 4 ) when padding near the corners. We pad channels of multi-channel data separately. Before padding, we make the data zero-mean by mean subtraction, which we found necessary for numerically stable Cholesky solves of AR coefficients and to meet the assumption of a zero-mean AR process. We add the mean back after padding. Depending on the extended neighborhood shape, we use a method based on covariance or a method based on autocorrelation. 2.1 Covariance method For one ( P=1 ) and two-pixel ( P=2 ) neighborhoods, the error to minimize can be expressed using the shorthand r_{ij} for the means of products that are elements of a covariance matrix r : \displaystyle\begin{aligned} P{=}1:\quad\text{MSE}=&{\textstyle\frac{1}{|S|}}\hskip-7.5pt\sum_{(y,x)\in S}\hskip-7.5pt\big(a_{1}I_{(y,x)+h_{1}}-I_{(y,x)+h_{0}}\big)^{2}=a_{1}^{2}r_{11}-2a_{1}r_{01}+r_{00}\\ P{=}2:\quad\text{MSE}=&{\textstyle\frac{1}{|S|}}\hskip-7.5pt\sum_{(y,x)\in S}\hskip-7.5pt\big(a_{1}I_{(y,x)+h_{1}}+a_{2}I_{(y,x)+h_{2}}-I_{(y,x)+h_{0}}\big)^{2}\\ =&\,a_{1}^{2}r_{11}+2a_{1}a_{2}r_{12}-2a_{1}r_{01}+a_{2}^{2}r_{22}-2a_{2}r_{02}+r_{00},\end{aligned} (5) \displaystyle\text{where }r_{ij}={\textstyle\frac{1}{|S|}}\hskip-7.5pt\sum_{(y,x)\in S}\hskip-7.5pt{I_{(y,x)+h_{i}}I_{(y,x)+h_{j}}},\quad r_{ij}=r_{ji}. (6) The LS solutions can be found by solving what are known as the normal equations : \displaystyle P{=}1: \displaystyle\tfrac{\partial\text{MSE}}{\partial a_{1}}=2a_{1}r_{11}-2r_{01}=0\quad \displaystyle\Rightarrow\quad\hphantom{\Bigg\{}a_{1}=\tfrac{r_{01}}{r_{11}} (7) \displaystyle P{=}2: \displaystyle\begin{cases}\frac{\partial\text{MSE}}{\partial a_{1}}=2a_{1}r_{11}+2a_{2}r_{12}-2r_{01}=0\\[3.0pt] \frac{\partial\text{MSE}}{\partial a_{2}}=2a_{1}r_{12}+2a_{2}r_{22}-2r_{02}=0\end{cases} \displaystyle\Rightarrow\quad\begin{cases}a_{1}=\tfrac{r_{01}r_{22}-r_{02}r_{12}}{r_{11}r_{22}-r_{12}^{2}}\\[10.0pt] a_{2}=\tfrac{r_{02}r_{11}-r_{01}r_{12}}{r_{11}r_{22}-r_{12}^{2}}.\end{cases} In implementation, we used a safe version of the division operator and its derivatives that replaces infinities with zeros in results. For any P ,

Read the complete text