Fundamentals of Regression (Chapter 2)
Miguel A. Mendez von Karman Institute for Fluid Dynamics · 2025
Read Fundamentals of Regression (Chapter 2) by Miguel A. Mendez von Karman Institute for Fluid Dynamics (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01920.txt.
Introduction
Regression is the branch of machine learning that finds relationships between variables, predicting one quantity from others. Miguel A. Mendez's 2025 chapter, the second in a larger work, opens with a review of the classic tools of the trade, written for readers coming from the natural sciences and engineering.
The chapter situates those tools inside scientific machine learning, where data-driven methods are combined with physical models and principles. Readers get the standard regression machinery first, presented as the foundation for the more advanced methods that follow. As a reference, it serves scientists who want learning algorithms grounded in the subjects they already know.
Catalog record
| CREATOR | Miguel A. Mendez von Karman Institute for Fluid Dynamics |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fundamentals of Regression (Chapter 2) Miguel A. Mendez von Karman Institute for Fluid Dynamics Abstract This chapter opens with a review of classic tools for regression, a subset of machine learning that seeks to find relationships between variables. With the advent of scientific machine learning 1 1 1 The term “scientific machine learning” refers to the integration of machine learning with models, principles, and data arising from the natural sciences and engineering. It does not imply that other forms of machine learning are non-scientific. Instead, it highlights a focus on problems where physical laws (e.g., conservation laws, PDEs, thermodynamics) play a central role. Scientific machine learning typically involves combining data-driven methods with domain knowledge, physics-based modeling, numerical simulation, and uncertainty quantification. The emphasis is on developing algorithms that are constrained by—or informed by—scientific theory, enabling improved prediction, interpretability, and generalization in complex physical systems. this field has moved from a purely data-driven (statistical) formalism to a constrained or “physics-informed” formalism, which integrates physical knowledge and methods from traditional computational engineering. In the first part, we introduce the general concepts and the statistical flavor of regression versus other forms of curve fitting. We then move to an overview of traditional methods from machine learning and their classification and ways to link these to traditional computational science. Finally, we close with a note on methods to combine machine learning and numerical methods for physics How to cite this work ⬇ 1 @InCollection { Mendez2024 , 2 author = { Mendez , Miguel A .}, 3 title = { Fundamentals of Regression }, 4 booktitle = { Machine Learning for Fluid Dynamics }, 5 editor = { Mendez , Miguel A . and Parente , Alessandro }, 6 publisher = { von Karman Institute }, 7 year = {2024}, 8 chapter = {2}, 9 isbn = {978-2875162090} 10 } 1 A note on notation and style Vectors, Matrices and lists . We use lowercase letters for scalar quantities, i.e. a\in\mathbb{R} . Bold lowercase letters are used for vectors, i.e., \bm{a}\in\mathbb{R}^{n_{a}} . The i-th entry of a vector is denoted with a subscript as \bm{x}_{i} or with Python-like notation as \bm{x}[i] . We use square brackets to create vectors from a set of scalars, e.g \bm{a}=[a_{0},a_{1},\dots a_{n_{a}-1}]\in\mathbb{R}^{n_{a}} . Unless otherwise stated, a vector is a column vector. The use of transposition when defining a vector embedded within the text of a paragraph or sentence (inline) is omitted. We use the upper case bold letters for matrices, e.g., \bm{A}\in{\mathbb{R}^{n_{r}\times n_{c}}} , with n_{r} the number of rows and n_{c} the number of columns. The matrix entry at the i-th row and j-th column are identified as \bm{A}_{i,j} or with the Python-like notation as \bm{A}[i,j] . When the Python notation is used, the indices begin with 0 . We occasionally work with lists of quantities. Following a Python notation, we enclose lists within round brackets and use bold letters for a list of vectors or matrices, e.g. \bm{\Gamma}_{*}=(\bm{x},\bm{y}) . Functions and Calculus . Lowercase letters followed by parentheses indicate functions, regardless of whether these are scalar or vector-valued functions, i.e. a(\bm{x}):\mathbb{R}^{n_{i}}\rightarrow\mathbb{R}^{n_{o}} . In a vector value function, \bm{y}=f(\bm{x}) and the subscript is used to define the mapping to each component, i.e. \bm{y}_{i}=f_{i}(\bm{x}) . The partial derivative of a function with respect to the input variables \bm{x}_{i} is denoted with the compact notation \partial_{x_{i}}f . The total derivative of a function f(\bm{x}) , with f:\mathbb{R}^{n_{I}}\rightarrow\mathbb{R}^{n_{o}} is denoted as {df}/{d\bm{x}}\in\mathbb{R}^{n_{i}\times n_{o}} or with the short-hand notation d_{\bm{x}}f\in\mathbb{R}^{n_{i}\times n_{o}} . The partial derivative of vector valued functions are collected produce the Jacobian \mathbf{d}f/\mathbf{dx} . The entries of the Jacobian are computed as \mathbf{d}f/\mathbf{dx}_{i,j}=\partial_{\bm{x}_{j}}f_{i} . In the case of a function f:\mathbb{R}^{n_{i}}\rightarrow\mathbb{R} , this becomes a row vector and is called gradient . Many authors use the nabla symbol \nabla f for the gradient, but we do not make the distinction and treat it as a Jacobian. Parametric and nonparametric representation . For parametric functions f:\bm{x}\in\mathbb{R}^{n_{x}}\rightarrow\mathbb{R}^{n_{y}} that depend on parameters \bm{w}\in\mathbb{R}^{n_{w}} , we use the notation \bm{y}=f(\bm{x};\bm{w}) or \bm{y}=f(\bm{x}|\bm{w}) . This distinction is important for differentiating between parametric and non-parametric models. For example, we can write a parabola y=c_{2}x^{2}+c_{1}x+c_{0} as a parametric function y=f(x;\bm{c}) with \bm{c}=[c_{1},c_{2},c_{3}] . This notation implies that we have a script that will require in inputs both x and \bm{c} ; this is the essence of parametric modeling. However, we could put the focus on the data used for making those predictions. Assuming that the model parameters \bm{c} were inferred from a training dataset \bm{\Gamma_{*}}=(\bm{x}_{*},\bm{y}_{*}) , we might also write the parametric function as y=f(x|\Gamma_{*}) . This notation implies that our script will require input x and the training data \bm{\Gamma_{*}} ; this is the essence of non-parametric modelling. Figure 1 : General overview of the regression framework, pictorially illustrated for a scalar problem. Left: Two possible models fit the training data (black dots). A prediction is requested and a stochastic process has to be fitted to the data. This can be described as in ( 2 ) with a deterministic model for the mean and a stochastic model for the local distribution. 2 General Concepts Regression is a subset of statistics and machine learning and, in particular, a subset of supervised (or predictive) learning . The goal is to learn a mapping from a continuous variable \bm{x}\in\mathbb{R}^{n_{x}} to another continuous variable \bm{y}\in\mathbb{R}^{n_{y}} . This is in contrast to classification, where the output variable is categorical (i.e. it contains a finite set of classes/categories, such as "yes" or "no", "cat" or "dog", "laminar" or "turbulent"). Regression and classification share much of the general mathematical framework, and most of the algorithms used for one can be used for another with little to no modifications. The boundaries sometimes are so blurred that some regression methods (e.g. logistic regression) are de facto tools for classification. In its goal of identifying continuous functions from data, regression methods share some common grounds with other curve-fitting methods, such as interpolation or smoothing. However, the key difference is in the statistical roots: the variables \bm{x},\bm{y} have a stochastic nature and thus generate a random process (that is, a distribution of possible functions). Our goal is to fit a random process to the data. In most cases, and indeed for all problems discussed in this lecture, we would be satisfied with a prediction of the mean function and a confidence interval around that mean. Let us introduce the general formalism with the help of Figure 1 , which pictorially represents the problem for the case of a scalar-valued function to ease the graphical representation. Let us assume that a set of n_{*} training points (black dots) is available. In the most general high-dimensional scenario, these could be stored in two matrices: \bm{X}_{*}:=\begin{bmatrix}\bm{x}_{*1}\dots\\ \bm{x}_{**2}\dots\\ \vdots\\ \bm{x}_{n_{p}}\dots\end{bmatrix}\in\mathbb{R}^{n_{*}\times n_{x}}\quad\mbox{and}\quad\bm{Y}_{*}:=\begin{bmatrix}\bm{y}_{*1}\dots\\ \bm{y}_{*2}\dots\\ \vdots\\ \bm{y}_{n_{p}}\dots\end{bmatrix}\in\mathbb{R}^{n_{*}\times n_{y}}\,. (1) To compress the notation, let us store this training data into a list \bm{\Gamma_{*}}=(\bm{X}_{*},\bm{Y}_{*}) . The simplest model to picture a stochastic process is an additive model: \bm{Y}(\bm{X}|\bm{\Gamma_{*}})=\tilde{f}(\bm{X}|\bm{\Gamma_{*}})+\mathcal{E}(\bm{X}|\bm{\Gamma_{*}})\,, (2) with \bm{X}\in\mathbb{R}^{n_{*}\times n_{x}} an arbitrary set of input points and \bm{Y}\in\mathbb{R}^{n_{*}\times n_{y}} the associated predictions. The first term \tilde{f}() is a deterministic function that provides one output given one input. The second term \mathcal{E}() is a random variable to which we associate a probability