Transfer learning RGB models to hyperspectral images with trainable tensor decompositions Mariette Schönfeld, Laurens Devos, Wannes Meert, Hendrik Blockeel Abstract Transfer learning makes it possible to use large vision networks on a variety of domains, by specializing their models’ general filters to new tasks. However, these networks assume the input images to have 3 input channels, making them incompatible with multi- or hyperspectral images. Current approaches that mitigate this incompatibility sacrifice information in either the image, or the model. This work proposes a novel approach that preserves the image and spatial information present in the model by using partially trainable tensor decompositions. We create such decompositions of pretrained convolutional filters, separating the filters into spatial and spectral components. The spectral components are then replaced with trainable components of higher channel dimensionality. This creates hyperspectral filters that can specialize to new datasets, while retaining the spatial patterns of the original filter. Experiments on a variety of hyperspectral datasets show that our approach is more accurate and robust than other hyperspectral transfer learning methods. 1 Introduction Hyperspectral images are images where each pixel is described by the intensities of many wavelengths (possibly hundreds), within or beyond the visible spectrum, rather than the three RGB values typically used. The additional information in this description makes no difference to the human eye, but can be crucial for data analysis. Hyperspectral images have proven valuable for a variety of complex vision tasks, including satellite imagery [ 37 ] , medical imaging [ 18 ] , and horticulture [ 19 ] . Hyperspectral image processing can roughly be categorized into remote sensing and near-range image processing. In remote sensing [ 37 ] , individual pixels are typically classified based on their own spectrum, and downstream tasks such as segmentation make use of that information. Near-range image processing [ 18 , 19 ] is more similar to classical vision in the sense that spatial patterns (such as those learned by convolutional neural networks) are often relevant. In this paper, we mostly focus on this second type. A challenge in training models on hyperspectral images is the acquisition of large amounts of hyperspectral data: this is expensive, time-consuming, and logistically challenging. As a result, datasets are typically small, making hyperspectral imaging an excellent candidate for transfer learning [ 25 ] . However, transfer learning requires a backbone model to transfer knowledge from, and a hyperspectral backbone is unlikely to be developed soon, for two reasons. First, the potential of transfer learning greatly depends on the variety and amount of data that the backbone model has seen. The most versatile and accurate backbones owe their success to their universally applicable filters, a result of training extensively on large, diverse datasets like ImageNet [ 3 ] . Because the amount and variety of hyperspectral data is significantly lower than for RGB images, creating backbone models of comparable quality for hyperspectral data is difficult. Second, the lack of a common data format makes the existence of a hyperspectral backbone impractical. Hyperspectral datasets vary in what wavelengths they use (and how many), whereas RGB datasets always use the same three wavelengths. Applying transfer learning in hyperspectral imaging therefore requires bridging varying input dimensionalities. For example, building a classification model for 50-dimensional inputs by transferring a 3-dimensional RGB backbone model necessitates addressing the input-channel gap. Several solutions have been proposed for solving this mismatch between hyperspectral data and current backbones. A lightweight solution is to perform dimensionality reduction on the spectral (i.e. channel or color) dimension of the image, in order to create a three-channel extract [ 5 , 16 ] . This sacrifices information in the image, but preserves the backbone. A different approach is to replace the first convolutional layer of the backbone by one with more input channels, while keeping the rest of the network structure intact [ 28 ] ; during fine-tuning, the new layer is then trained from scratch. This approach allows the new model to use information across all channels in the image, but has the downside that the backbone’s first-layer filters are sacrificed. Furthermore, this also increases the number of trainable parameters and therefore the risk of overfitting to spurious correlations grows. This paper proposes a novel approach for tackling the above-mentioned mismatch. It is known that weights of backbone models can be compressed with minimal performance loss using tensor decompositions [ 4 ] because of the relative simplicity of the learned filters [ 35 ] . These techniques are typically used with the goal of model compression, but tensor decompositions have another advantage: by design, they decouple spatial and spectral patterns [ 13 ] . This paper builds on this property to facilitate transfer learning from 3-channel RGB backbones for hyperspectral tasks. We accomplish this by decomposing the convolutional filters in the first layer, separating the filters into their spatial and spectral components. Each 3-channel spectral component is then replaced by a higher-dimensional, hyperspectral one. Decompressing now provides a filter that can be convolved with the hyperspectral image, and contains the spatial patterns of the original filter. The new components can then be trained in a standard transfer learning pipeline. This solves the dimensionality mismatch between the hyperspectral image and the backbone model, preserves both the model and the image, and enables a traditional transfer learning pipeline for hyperspectral images with a minimal number of trainable parameters. We first provide background in section 2 on transfer learning and tensor decompositions. Then, we introduce our method in section 3 . Section 4 covers related work with regards to transfer learning in hyperspectral imaging and tensor decompositions in neural networks. Finally, we show through experiments in section 5 how our method is more accurate and more robust against overfitting than other hyperspectral transfer learning approaches. 2 Background 2.1 Transfer learning Transfer Learning is a collection of techniques that involve the transfer of previously gained knowledge to a new task [ 25 ] . This can be as simple as starting the learning process from pretrained weights instead of random initializations with the goal accelerating convergence. In vision contexts however, it can be difficult to prevent overfitting in low-data scenarios and so it is oftentimes recommended [ 26 ] to freeze the feature extraction (i.e. convolutional) stage of a network, and only train the classification layers. This difference leads to a distinction we make in this work: a pretrained model versus a backbone model. In hyperspectral images, it is common to pretrain a model on related datasets [ 31 ] , but this is unlikely to result in a model so versatile that transferring from a remote sensing task to a medical task can be done by just retraining the classification layer. With a backbone model, we specifically refer to a model that has been pretrained on large volumes of diverse data (like ImageNet), and can be applied to a variety of vision tasks with minimal retraining. Such backbone models are widely available for regular vision, but do not exist for hyperspectral vision. One interesting observation of such backbone models is that the filters in their first layer tend to exhibit colored, elliptical shapes, which cause edge-detection-like behavior [ 35 ] . Such filters contain both spatial and spectral information. With spatial we mean the size, orientation, and relative location of the shapes, and with spectral we mean their colors. Although this behavior is not explicitly induced in the network, the fact that it seems to happen for all backbone models gives the impression that it is important and should be preserved for robust and accurate vision. 2.2 Tensor decompositions for convolutional filters Tensor decompositions are a collection of techniques for approximating complex tensors with smaller, simpler tensors [ 13 ] . Many decompositions are higher-order generalizations of matrix decompositions, where order refers to the number of indices in the array. A vector is a first order tensor, a matrix a second order, a cube a third order tensor, e.t.c. Many matrix operations have tensor generalizations, like outer products, or tensor multiplication. Although matrix decompositions can be applied to tensors by unfolding them to a matrix, tensor decompositions can preserve the higher-order structure of the tensor by using components for each order. For example, an image can be seen as a third-order tensor with one channel dimension and two spatial dimensions, and a decomposition could involve three projection matrices, each acting on its own order. Matrix decompositions were already of interest to deep learning researchers as a tool for network compression, but tensor decompositions were more appealing for two reasons [ 21 ] : they offer the opportunity to rewrite the convolutional operator directly over the components of the decomposition instead of having to decompress the filters. This makes tensor decomposed layers not just more efficient for storage, but also for inferencing [ 11 , 12 , 17 ] . Second, it has been observed that the aforementioned elliptical shapes in backbone filters have high similarity to either low-rank or multilinear patterns [ 4 ] , the mathematical structures that tensor decomposition approximate tensors with. As there are multiple types of decompositions and ways to apply them to a tensor, there is also not a single universal method to use them for model compression. 3 Filter decompositions for hyperspectral transfer learning The previous sections can be summarized as follows: hyperspectral vision could benefit from transfer learning. This is hindered by the lack of hyperspectral backbone models and the dimensionality mismatch between hyperspectral images and 3-channel backbone models. Such backbone models typically exhibit colored, elliptical shapes, which should ideally be preserved. These filters lend themselves well to tensor decomposed convolutions, which split the filters into spatial and spectral components. We will use this split to solve the mismatch between backbone models and hyperspectral images. Consider a backbone model. Typically the first layer is a 2D convolution with a fourth order weight tensor W^{1} of shape C_{out}\times C_{in}\times k_{1}\times k_{2} with C_{out} the number of filters or output channels, C_{in}=3 the input channels, and k_{1} and k_{2} the height and width of the filters. An input image needs to match the number of input channels C_{in} and so W^{1} cannot be convolved with a hyperspectral image X of shape \hat{C}_{in}\times H\times W because \hat{C}_{in}>3 . We approximate the filters in W^{1} with tensor decompositions, providing us with spatial and 3-channel spectral components. We can then replace the spectral components with larger ones that match \hat{C}_{in} . Decompressing the decomposition would give C_{out} filters of shape \hat{C}_{in}\times k_{1}\times k_{2} , which contain the spatial patterns of the original filter but can be convolved with X . A transfer learning pipeline can then be used for learning informative values for the spectral components. We discuss this approach (visualized in figure 1 ) for CP and Tucker decompositions, the two most popular decompositions [ 13 ] . Figure 1 : Illustration of our general approach. Values for the new hyperspectral component can be learned in a transfer learning pipeline, while keeping the spatial components frozen. Figure 2 : Illustration of how a filter is decomposed into its spectral ( c_{o}^{i} ) and spatial ( x_{o}^{i} and y_{o}^{i} ) components with a CP decomposition. 3.1 CP decompositions CP (Canonic Polyadic) decompositions approximate tensors as a finite sum of rank-one tensors . A rank-one tensor is a tensor that can be written as the outer product of vectors (illustrated in figure 2 ). This makes a second order rank-one tensor the same as a separable filter [ 23 ] , and a third order rank-one tensor can be seen as a colored separable filter with two spatial vectors, and one spectral vector indicating the color. Changing the values in this third vector only changes the colors in the filter, but not the shapes. The rank of a decomposition then refers to how many of such filters can be part of the summation. The illustration of this decomposition in figure 2 demonstrates how changing the values in the spectral vectors only affects the colors in the filter, not the shapes. We approximate a filter W_{o}^{1} with CP decomposition of rank R as: W_{o}^{1}\approx\overline{W_{o}^{1}}=\sum_{r=0}^{R}c_{o,r}\circ x_{o,r}\circ y_{o,r},\ 0\leq o