Statistical Mechanics of Support Vector Regression

Abdulkadir Canatar, SueYeon Chung · 2024

Read Statistical Mechanics of Support Vector Regression by Abdulkadir Canatar, SueYeon Chung (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05439.txt.

Introduction

Abdulkadir Canatar and SueYeon Chung apply physics methods to a classic learning algorithm in this two thousand twenty-four paper. The algorithm is support vector regression, used to decode signals such as neural activity. The authors treat the learning problem as a physical system and compute its average behavior. Their focus is how geometry of the data shapes decoding precision. Neural noise plays a central role in the analysis.

They find a sharp phase change in training error as data load grows. The balance between a tolerance setting and noise drives the shift. Test error shows a double descent, falling, rising, then falling again. These patterns mirror puzzles seen in deep learning. The work links neuroscience decoding to the theory of learning curves.

Catalog record

CREATORAbdulkadir Canatar, SueYeon Chung
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05439

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Statistical Mechanics of Support Vector Regression Preprint: APS/123-QED Abdulkadir Canatar, SueYeon Chung Abstract A key problem in deep learning and computational neuroscience is relating the geometrical properties of neural representations to task performance. Here, we consider this problem for continuous decoding tasks where neural variability may affect task precision. Using methods from statistical mechanics, we study the average-case learning curves for ε \varepsilon -insensitive Support Vector Regression ( ε \varepsilon -SVR) and discuss its capacity as a measure of linear decodability. Our analysis reveals a phase transition in the training error at a critical load, capturing the interplay between the tolerance parameter ε \varepsilon and neural variability. We uncover a double-descent phenomenon in the generalization error, showing that ε \varepsilon acts as a regularizer, both suppressing and shifting these peaks. Theoretical predictions are validated both on toy models and deep neural networks, extending the theory of Support Vector Machines to continuous tasks with inherent neural variability. Introduction.— Modern machine learning applications often require the ability to perform tasks that remain invariant under certain transformations of input data. For instance, decoding of a class should remain unaffected by variations in the object’s representations due to factors like translations LeCun et al. 2015 . Deep neural networks achieve this by learning a meaningful representation of input data, from which decoding the information relevant to the task becomes easier. For instance, an effective representation’s decoding should be invariant to task-irrelevant transformations while remaining covariant with respect to task-relevant information. Understanding such geometric properties of neural representations is essential for unraveling the computational mechanisms implemented by neural networks Bengio et al. 2013 ; Achille and Soatto 2018 ; Chung and Abbott 2021 . Previous studies have explored the geometrical properties of neural representations in relation to invariant prediction, particularly focusing on the linear decodability under neural variability Chung et al. 2018 ; Chung and Abbott 2021 ; Wakhloo et al. 2023 . However, these studies largely concentrate on classification tasks, despite the fact that decoding continuous variables, such as an object’s position or orientation, is essential in fields like control and robotics where precise and continuous outputs are critical. This leaves a gap in our understanding of how effective decoding of continuous variables can be achieved in the presence of neural variability. In this letter, we study \varepsilon -insensitive Support Vector Regression ( \varepsilon -SVR) algorithm Drucker et al. 1996 ; Schölkopf et al. 2000 to analyze continuous tasks with input data variances. The \varepsilon -SVR task is similar to the ridge regression, except that the objective is satisfied when the prediction error lies within a tube of size \varepsilon (Fig. 1 ). Here, the tube size \varepsilon is a hyperparameter that defines the margin of tolerance for input deviations and needs to be fine-tuned. Using the replica method Mézard et al. 1987 ; Engel 2001 ; Mezard and Montanari 2009 , we develop an analytical theory of generalization in \varepsilon -SVR that relates the geometry of invariances to the precision of decoding a continuous variable. Our theory recovers the previous results on ridge regression in the limit \varepsilon\to 0 Bordelon et al. 2020 ; Jacot et al. 2020 ; Canatar et al. 2021 ; Loureiro et al. 2021 ; Simon et al. 2023 ; Atanasov et al. 2022 ; Atanasov et al. 2024 . Theoretical Framework.— We assume a supervised learning setup where the input data is represented by N -dimensional center representations \bar{\bm{\psi}} , and the labels \bar{y} are linearly generated by a coding direction \bar{\mathbf{w}} , i.e. \bar{y}=\bar{\mathbf{w}}\cdot\bar{\bm{\psi}} . Here, centers \bar{\bm{\psi}} are treated as random variables and capture the task-relevant information from the input data. Figure 1: Decoding the angle of sticks from their images as an example of invariant regression. Blue regions depict the encoding representations in (\psi_{1},\psi_{2}) -space induced by nuisance variances for each angle with varying geometries. They depict possible noise geometries with {\bm{\delta}} parallel (left), orthogonal (middle), and partially aligned (right) with the coding direction \bar{\mathbf{w}} . In contrast, the algorithm receives encoding representations {\bm{\psi}} which include both the task-relevant centers \bar{\bm{\psi}} and task-irrelevant variations modeled by random variables {\bm{\delta}} , and assumed to be of the form {\bm{\psi}}\equiv\bar{\bm{\psi}}+{\bm{\delta}} . The noise representations {\bm{\delta}} represent nuisance variations around the centers. The goal is to learn a linear predictor y={\mathbf{w}}\cdot{\bm{\psi}} . At each instantiation of a training set, P -samples \{\bar{\bm{\psi}}^{\mu},{\bm{\psi}}^{\mu}\}_{\mu=1}^{P} are drawn i.i.d. from their respective distributions, and labels \bar{y}^{\mu} and predictions y^{\mu} are generated. Due to the additional noise, the predictions always mismatch with the labels when P>N and overfit to noise when P<N . To account for this variance, we consider the \varepsilon -SVR algorithm \displaystyle\min_{{\mathbf{w}}\in\mathbb{R}^{N}}\frac{1}{2}\norm{\w}^{2}+\frac{1}{2\lambda}\sum_{\mu=1}^{P}\left[\absolutevalue{y^\mu-\bar y^\mu}-\varepsilon\right]_{+}^{2} (1) where \varepsilon is the tube size, \lambda is the ridge parameter and \left[x\right]_{+}\equiv\max(0,x) . We note that both \lambda and \varepsilon act as a regularizer for regression since both control the effect of training error in the final solution. However, we found that they qualitatively have different effects on generalization (see SI. A ). While our calculation is valid for arbitrary \lambda and \varepsilon , for our purposes we only consider the ridgeless case ( \lambda=0 ) which enforces vanishing training error. Our goal is to compute the average-case training and generalization errors for \varepsilon -SVR. By treating the training set as a quenched disorder, we compute these average quantities using the replica method of statistical physics and consider the following partition function \displaystyle\overline{\log Z}=\overline{\log\int d{\mathbf{w}}e^{-NS({\mathbf{w}},{\bm{\psi}}^{\mu},\bar{\bm{\psi}}^{\mu})+J\mathcal{O}({\mathbf{w}})}} (2) \displaystyle S=\frac{1}{2N}\norm{\w}^{2}+\frac{1}{2\lambda N}\sum_{\mu=1}^{P}\left[\absolutevalue{\w\cdot\bpsi^\mu- \bar\w\cdot\bar\bpsi^{\mu}}-\varepsilon\right]_{+}^{2}, where the overline denotes the quenched average over the training set \{{\bm{\psi}}^{\mu},\bar{\bm{\psi}}^{\mu}\} of size P . In the thermodynamic limit P,N\to\infty while keeping \alpha\equiv P/N\sim\mathcal{O}(1) , the free energy S in Eq. 2 becomes self-averaging and concentrates around the solution to the problem in Eq. 1 . Then, the average of any observable \braket{\mathcal{O}({\mathbf{w}})} can be computed by taking derivatives of \overline{\log Z} with respect to source J . The training error E_{tr} and generalization error E_{g} , the two observables we are interested in, are given by \displaystyle E_{tr} \displaystyle=\overline{\frac{1}{P}\sum_{\mu=1}^{P}\left[\absolutevalue{\w\cdot\bpsi^\mu- \bar\w\cdot\bar\bpsi^{\mu}}-\varepsilon\right]_{+}^{2}} \displaystyle E_{g} \displaystyle=\overline{\Braket{\left({\mathbf{w}}\cdot{\bm{\psi}}-\bar{\mathbf{w}}\cdot\bar{\bm{\psi}}\right)^{2}}_{{\bm{\psi}},\bar{\bm{\psi}}}}. (3) Replica Calculation.— For simplicity, we consider a joint Gaussian distribution of center and encoding representations as \displaystyle\begin{bmatrix}{\bm{\psi}}\\ \bar{\bm{\psi}}\end{bmatrix}\sim\mathcal{N}\left(0,\begin{bmatrix}{\bm{\Sigma}}_{\bm{\psi}}&{\bm{\Sigma}}_{{\bm{\psi}}\bar{\bm{\psi}}}\\ {\bm{\Sigma}}_{{\bm{\psi}}\bar{\bm{\psi}}}^{\top}&{\bm{\Sigma}}_{\bar{\bm{\psi}}}\end{bmatrix}\right). (4) Note that the solution to the objective in Eq. 2 under this data distribution depends highly on the correlation between centers \bar{\bm{\psi}} and encoding representations {\bm{\psi}} . Specifically, in the limit P\to\infty , optimal weights and the generalization error asymptotes to: \displaystyle E_{\infty}=\tr\bar{\mathbf{w}}\bar{\mathbf{w}}^{\top}{\bm{\Sigma}}_{\bar{\bm{\psi}}}-\tr{\mathbf{w}}_{*}{\mathbf{w}}_{*}^{\top}{\bm{\Sigma}}_{{\bm{\psi}}},\;{\mathbf{w}}_{*}={\bm{\Sigma}}_{\bm{\psi}}^{-1}{\bm{\Sigma}}_{{\bm{\psi}}\bar{\bm{\psi}}}\bar{\mathbf{w}}, where \tr denotes the normalized trace operator, {\mathbf{w}}_{*} is the target weights projected to the learnable subspace, and E_{\infty} is the irreducible error corresponding to the unexplainable variance in the target Atanasov et al. 2022 . Performing the quenched average with this distribution under the replica symmetric ansatz and a saddle point approximation (see SI. A for details), the mean-field solution is described by three effective quantities: the effective regularization \tilde{\lambda} , effective tube size \tilde{\varepsilon} and the effective load \tilde{\alpha} : \displaystyle\tilde{\lambda} \displaystyle=\lambda+\mathcal{F}(\tilde{\lambda},\tilde{\varepsilon}),\quad\tilde{\varepsilon}=\varepsilon\big/\sqrt{\mathcal{G}(\tilde{\lambda},\tilde{\varepsilon})},\quad\tilde{\alpha}=\alpha f\left(\tilde{\varepsilon}\right) (5) where \tilde{\lambda} and \tilde{\varepsilon} obey the coupled self-consistent equations \displaystyle\mathcal{F}(\tilde{\lambda},\tilde{\varepsilon}) \displaystyle=\tr{\bm{\Sigma}}_{\bm{\psi}}{\mathbf{G}}^{-1},\;\;{\mathbf{G}}={\mathbf{I}}+\frac{\tilde{\alpha}}{\tilde{\lambda}}{\bm{\Sigma}}_{\bm{\psi}},\;\;\gamma=\partial_{\tilde{\lambda}}\mathcal{F}(\tilde{\lambda},\tilde{\varepsilon}) \displaystyle\mathcal{G}(\tilde{\lambda},\tilde{\varepsilon}) \displaystyle=\frac{E_{\infty}+\tr{\mathbf{w}}_{*}{\mathbf{w}}_{*}^{\top}{\bm{\Sigma}}_{\bm{\psi}}{\mathbf{G}}^{-2}}{1-g(\tilde{\varepsilon})\gamma} (6) where the functions f(\tilde{\varepsilon}) and g(\tilde{\varepsilon}) are defined as \displaystyle f(\tilde{\varepsilon}) \displaystyle=\Braket{\theta\left(\absolutevalue{z}-\tilde{\varepsilon}\right)}_{z},\;\;g(\tilde{\varepsilon})=\frac{\Braket{\left(\absolutevalue{z}-\tilde{\varepsilon}\right)^{2}\theta\left(\absolutevalue{z}-\tilde{\varepsilon}\right)}_{z}}{\Braket{\theta\left(\absolutevalue{z}-\tilde{\varepsilon}\right)}_{z}} where \braket{f(z)}_{z} denotes integration over z with respect to standard normal distribution. Furthermore, the function \mathcal{G}(\tilde{\lambda},\tilde{\varepsilon}) coincides with the generalization error defined in Eq.

Read the complete text