Statistical Methods and Modal Decompositions for Gridded and Scattered Data: Meshless Statistics and Meshless Data Driven Modal Analysis Miguel A. Mendez, Manuel Ratz, Damien Rigutto von Karman Institute for Fluid Dynamics Abstract Statistical tools are crucial for studying and modeling turbulent flows, where chaotic velocity fluctuations span a wide range of spatial and temporal scales. Advances in image velocimetry, especially in tracking-based methods, now allow for high-speed, high-density particle image processing, enabling the collection of detailed 3D flow fields. This lecture provides a set of tutorials on processing such datasets to extract essential quantities like averages, second-order moments (turbulent stresses) and coherent patterns using modal decompositions such as the Proper Orthogonal Decomposition (POD). After a brief review of the fundamentals of statistical and modal analysis, the lecture delves into the challenges of processing scattered data from tracking velocimetry, comparing it to traditional gridded-data approaches. It also covers research topics, including physics-based Radial Basis Function (RBF) regression for meshless computation of turbulent statistics and the definition of an RBF inner product, which enables meshless computation of data-driven decompositions. These include traditional methods like Proper Orthogonal Decomposition (POD) and Dynamic Mode Decomposition (DMD), as well as advanced variants such as Spectral POD (SPOD) and Multiscale POD (mPOD). We refer to this framework as "Meshless Data Driven Decomposition". Six exercises in Python are provided. All codes are available at this github repository. NOTE 1: This lecture is an updated version of the lecture “Statistical Treatment, Fourier, and Modal Decomposition", which was given in the previous edition of the lecture series and can be found in https://arxiv.org/abs/2201.03847 . Here the session on traditional modal decompositions has been shortened to give more space to the idea of meshless computation of statistics and data-driven decompositions. NOTE 2: These lecture notes are now being expanted into a book, covering statistics, traditional (gridded) modal analysis, constrained RBF and meshless modal analysis. Stay tuned for more :) ! A note on notation and style Vectors, Matrices and lists . We use lowercase letters for scalar quantities, i.e. a\in\mathbb{R} . Bold lowercase letters are used for vectors, i.e. \bm{a}\in\mathbb{R}^{n_{a}} . The i-th entry of a vector is denoted with a subscript as \bm{x}_{i} or with Python-like notation as \bm{x}[i] . Unless otherwise stated, a vector is a column vector. Thus, we omit the use of transposition when defining a vector embedded within the text of a paragraph or sentence (inline). We use upper case bold letters for matrices, e.g. \bm{A}\in{\mathbb{R}^{n_{r}\times n_{c}}} , with n_{r} the number of rows and n_{c} the number of columns. The matrix entry at the i-th row and j-th column are identified as \bm{A}_{i,j} or with the Python-like notation as \bm{A}[i,j] . When Python notation is used, the indices begin with 0 . We use square brackets to create vectors from a set of scalars, e.g \mathbf{a}=[a_{0},a_{1},\dots a_{n_{a}-1}]\in\mathbb{R}^{n_{a}} or matrices from a set of vectors, e.g. \mathbf{U}=[\bm{u}_{0},\bm{u}_{1}\dots u_{n_{t}-1}]\in\mathbb{R}^{n_{u}\times n_{t}} . Note the difference between bold bold italicized \bm{a} or \bm{U} and upright bold \mathbf{a} or \mathbf{U} . The first is used for quantities that are vector-valued or matrices by their nature (e.g. velocity fields or the Reynolds stresses), that is, even in their continuous form. The second is used for sampled quantities. Hence, a collection of PTV snapshots is denoted as \bm{U} , while the Reynolds stress tensor is denoted as \mathbf{R} . Sampling The sampling of a continuous function is stored in a vector or a matrix. We assume uniform sampling in both space and time. For vector \mathbf{d}(t) sampled the time domain t , considering a discretization \mathbf{t_{k}}=k\Delta t , with f_{s}=1/\Delta t the sampling frequency and k=[0,1,\dots n_{t}-1] , we could write \mathbf{d}[k] or \mathbf{d}_{k} or \mathbf{d}(\mathbf{t_{k}}) . The same is true for the space domain, although a matrix linear index must be introduced. This is important when we transform a matrix (for example, a spatial realization of a quantity) into a vector. For example, let p[i,j] be the 2D discretization of a pressure field p(x,y) , where the axes were discretized as i\in[0,n_{x}-1] and j\in[0,n_{y}-1] . For this lecture, that field would be written as a single "snapshot" vector \bm{p}\in\mathbb{R}^{n_{s}} , with n_{s}=n_{x}n_{y} . The entries in this vector would be accessed with a linear matrix index, denoted in bold, i.e. \bm{p}[\bm{i}] . The way this accesses the data in the matrix depends on whether the flattening is performed column-wise or row-wise. For example, for a matrix \bm{A}\in\mathbb{R}^{3\times 3} , the column-wise and row-wise matrix indices are 1 1 1 Recall that here use Python-like indexing. Hence, the first entry is 0 and not 1 \mbox{column-wise }\bm{i}:\,\,\bm{A}=\begin{bmatrix}0&3&6\\ 1&4&7\\ 2&5&8\end{bmatrix}\quad\mbox{row-wise }\bm{i}:\,\,\bm{A}=\begin{bmatrix}0&1&2\\ 3&4&5\\ 6&7&8\end{bmatrix}\,,. 1 Introduction and lecture format Statistical tools for describing space-time correlations, measuring turbulent stress, and identifying coherent patterns are essential in experimental fluid mechanics Pope, (2000) ; Tennekes, (1972) ; Davidson, (2004) . In the past decade, these tools have advanced to match rapid developments in image velocimetry and increasing spatial and temporal resolution. A significant advancement has been the adoption of 3D tracking velocimetry over traditional correlation-based methods ( Schanz et al.,, 2016 ; Tan et al.,, 2020 ) , which produce scattered data and introduce new challenges for post-processing, such as calculating mean fields and Reynolds stresses. The current standard approach involves interpolating scattered data onto a uniform grid to facilitate traditional analyses. For instantaneous fields—such as those required for computing derivatives or pressure reconstruction—methods like Vic+, Vic#, FlowFit ( Schneiders and Scarano,, 2016 ; Scarano et al.,, 2022 ; Schanz et al.,, 2016 ) , and Meshless Track Assimilation ( Sperotto et al., 2024b, ) incorporate physics-based constraints (e.g., divergence-free conditions) to ensure robust interpolation. For statistical computations, such as mean fields and Reynolds stresses, techniques like binning and ensemble PTV (EPTV, Discetti and Coletti, (2018) ) are widely used, dividing the domain into bins where local statistics are calculated ( Kähler et al.,, 2012 ) . With dense datasets, these methods often outperform cross-correlation techniques in computing Reynolds stresses ( Atkinson et al.,, 2014 ; Schröder et al.,, 2018 ) . This lecture provides an overview of statistical methods and modal decomposition techniques, covering traditional approaches for gridded data and new methods for scattered data. Although relevant literature is referenced as accurately as possible, this chapter is not intended as a comprehensive review of image velocimetry post-processing. Instead, it serves as a hands-on tutorial to guide the practical computation of key quantities, their physical significance, and their interpretation. Accordingly, the lecture is structured around a set of exercises in Python . Readers interested in learning Python for scientific computing are referred to Svein Linge, (2019) ; Langtangen, (2016) ; Johansson, (2018) ; Paul J. Deitel, (2019) . The lecture focuses on the analysis of two selected datasets, described in Section , along with instructions for downloading or data generation. Section 3 provides a concise review of fundamental statistical concepts, from first- and second-order moments to ergodicity, power spectral densities, and cross-coherency. Section 4 addresses the statistical treatment of turbulence, covering definitions of key quantities and analysis types (pointwise statistics vs. space/time analysis) and their computations. Section 7 shows how to compute these quantities for gridded or scattered data, introducing the concept of Physics-Constrained Radial Basis Function (RBF) regression and its application to the processing of scattered data Sperotto et al., (2022) ; Sperotto et al., 2024a () . Section 6 then focuses on modal decompositions, beginning with the continuous formulation of Proper Orthogonal Decomposition (POD) and proceeding to its computation for gridded data using matrix factorizations and for scattered data using physics-constrained RBF. Generalizations of this methodology to other popular decompositions, such as Dynamic Mode Decomposition (DMD, Schmid, (2010) ; Rowley et al., (2009) ) and Multiscale POD (mPOD, Mendez et al., (2019) ; Mendez, (2023) ), are also presented. 2 Collecting the datasets We consider two test cases in this lecture, described in the following subsections. 2.1 Transient flow past a cylinder This datasets consist of time-Resolved PIV for the flow past a cylinder of d=$5\text{\,}\mathrm{mm}$ diameter and L=$20\text{\,}\mathrm{cm}$ length in transient conditions, with varying free stream velocity. The experiments were carried out in the L10 low-speed wind tunnel of the von Karman Institute. The details of the experiment are described in Mendez et al., (2020) . The dataset can be downloaded by the provided script Get_Cylinder_DATA_PIV_PTV.py. This data set contains n_{t}=13200 velocity fields sampled at f_{s}=$3\text{\,}\mathrm{kHz}$ over a grid of 71\times 30 points. The cylinder has a diameter d=$5\text{\,}\mathrm{mm}$ and the spatial resolution is approximately \Delta x=$0.85\text{\,}\mathrm{mm}$ . The velocity of the free stream was varied from U_{\infty}\approx$12\text{\,}\mathrm{m}\text{/}\mathrm{s}$ to U_{\infty}\approx$8\text{\,}\mathrm{m}\text{/}\mathrm{s}$ , producing an evident change in the frequency of the vortex shedding from f\approx$459\text{\,}\mathrm{Hz}$ to f\approx$303\text{\,}\mathrm{Hz}$ . The Reynolds number is in the range Re\in[2600,4000] . Figure 1 shows a snapshot of the PIV velocity field on the top-left, together with the location of three probes P1, P2, P3, from which the signal will be analyzed in further details in the following exercises. The figure on the top right shows the evolution of the free stream velocity (probe P1). Figure 1 : Right: snapshot of the velocity field from the TR-PIV measurements considered in this test case (top) and synthetic scattered data simulating a PTV velocimetry. Left: time evolution of the free stream during the experiment, collected from Probe P1 (top) and power spectral density of the signal in probe P2 (bottom). Being time-resolved, this test case is well suited for testing the computation of power spectral densities, cross-coherency, and advection velocity between two points. Due to the change in free stream velocity and the associated variation in the vortex shedding, this also offers an excellent test case to benchmark the multiscale POD’s time-frequency localization capabilities, as extensively analyzed in Mendez et al., (2020) . Figure 1 shows, on the bottom right, the power spectral density from the time series extracted in probe P2. This test case has also been used in other lecture series, such as in Mendez, (2022) and Mendez, 2024a () , to compare the performances of various linear and nonlinear dimensionality reduction techniques. On the other hand, this experiment did not use tracking velocimetry. The provided code instead offers an efficient way to generate synthetic scattered data from gridded data, simulating a PTV acquisition. An example of a synthetic PTV field extracted from the PIV is shown in Figure 1 on the bottom left. It is important to emphasize that the goal here was not to precisely replicate the PTV evaluation process since a comparison between tracking-based and cross-correlation-based velocimetry is already addressed in the other provided test case. The objective is to test the tools developed in this lecture for processing time-resolved tracking. 2.2 A round jet flow The second dataset showcases a horizontal water jet generated by the Dantec Dynamics Educational-PIV system. The nozzle has an exit diameter of d = 5 cm and operates within an aquarium measuring 80 x 35 x 40 cm. The jet velocity is approximately 20\text{\,}\mathrm{mm}\text{/}\mathrm{s} , with the water pump set to 30% power. Images are captured using a camera with a resolution of 1920\times 1200 px at a frame rate of 160\text{\,}\mathrm{Hz} . The illumination is provided by a LED light source positioned at the bottom of the aquarium. As the system only supports time-resolved acquisition, a total of 10^{5} frames are recorded. These frames are down-sampled by a factor of 10 to obtain image pairs at a frequency of 16 Hz, with a time separation of 6.25 ms between frames, ensuring appropriate pixel displacement while avoiding unnecessary oversampling of the jet oscillation. The image pairs are processed using two methods. The first is a classical PIV approach is applied using the Particle Image Reconstruction Software (PAIRS) ( Astarita and Cardone,, 2004 ; Astarita,, 2006 ; Astarita,, 2007 ; Astarita,, 2008 ; Astarita,, 2009 ) . The second is an in-house PTV tracking approach with a custom two-pulse PTV algorithm. This is based on the super-resolution approach from Keane et al., (1995) and incorporates a predictor based on the PIV results. Particle positions were identified with an adapted version of the open-source TracTrac code from Heyman, (2019) . Figure 2 : Picture of the jet flow setup (left). Typical velocity vectors from the PTV analysis (right) PIV and PTV velocity fields are obtained with the Get_Jet_DATA_PIV_PTV.py script. The data includes n_{t}=10\ 000 time steps with a temporal resolution of \Delta t=$62.5\text{\,}\mathrm{ms}$ . For PIV analysis, velocity vectors are obtained using 64\times 64 px interrogation windows with 75\text{\,}\mathrm{\%} overlap, providing a vector pitch of \Delta x=\Delta y=1.83 mm. It is worth noting that, due to limitations in illumination, the PIV vector fields contain a significant number of NaNs (Not a Numbers). For the same reason, the PTV field are much sparser near the boundaries. However, this is not considered a major limitation for the didactic purposes of this exercise. To circumvent the issues, we reduce the region of interest to a field of 50\times$100\text{\,}\mathrm{mm}$ in x and y , resulting in 27 and 55 vectors in both axes. For PTV analysis, the algorithm extracts the velocity at the position of n_{p}\approx 150 particles for each time step in the reduced FOV. This data is provided as a large ensemble of scattered data. The large number of images and vectors for this second data set offers an excellent dataset for statistical analysis using PIV and PTV approaches. 3 Fundamentals of statistics This section reviews fundamental concepts in statics, mainly first—and second-order moments in section 3.1 , and then applies these to time series, with the important notion of Ergodicity in section 3.2 . Finally, we close with some tools for spectral analysis in 3.3 . 3.1 First and second order statistics In the statistical treatment of turbulent flows, we consider the velocity field as an infinitely dense field of random variables . We revisit this concept in Section 3 . For now, we focus on the two key properties we are primarily interested in. Let u represent a scalar random variable. For example, this could correspond to the value of one of the velocity components at a specific point in space. A random variable can be seen as a function that maps outcomes from a sample space of real numbers; we denote it as u\in\mathbb{R} in this case. This is the set of all possible outcomes for u . We treat it as a continuous variable because it can take any value in ]-\infty,\infty[ , as opposed to discrete random variables where only a discrete set of points can be sampled. Our continuous random variables are identified by a probability density function (pdf) f_{u}(u) that describes the likelihood of different outcomes. This pdf allows to assign the probability that u_{m}\leq u_{n}(\mathbf{t_{k}})0 , it means that the two variables tend to increase or decrease together, while \text{cov}(u,v)=0 means that the two variables are uncorrelated . This concept should not be confused with that of independence : we say that the variables are independent if f_{u,v}(u,v)=f_{u}(u)f_{v}(v) . Note that \text{var}(u)=\text{cov}(u,u) . One can thus normalize covariances using the variances of each variable. The result is the correlation coefficient , ranging between -1 and 1: R_{u,v}=\frac{\text{cov}(u,v)}{\sqrt{\text{var}(u)\text{var}(v)}} (5) All subsequent analysis builds on these definitions, as we will discuss shortly. However, it is important to emphasize that the practical computation of these quantities always involves approximations, as we never have an analytic expression for the PDFs involved. Thus, estimating these quantities requires constructing approximations of the underlying distributions based on a large number of samples. In practice, we rely on samples of the random variable which we collect in a vector \mathbf{u}\in\mathbb{R}^{n_{u}} . We thus must replace the notion of PDF with that of probability mass function (PMF). With this, we estimate the probability of having a specific outcome as p(u_{k})=n_{k}/n_{u} , with n_{k} the number of times the specific value u_{k} occurs and n_{u} the total number of samples corrected. The integrals in ( 2 ) and ( 3 ) should are then replaced with summations: \displaystyle\mu_{u} \displaystyle\approx\mathbb{E}_{\sim\mathbf{u}}\{\mathbf{u}\}=\tilde{\mu}_{u}=\sum_{k}u_{k}p(u_{k})=\frac{1}{n_{u}}\sum^{n_{u}}_{k=1}u_{k} (6) \displaystyle\sigma^{2}_{U} \displaystyle\approx\tilde{\mbox{var}}\{\mathbf{u}\}=\sum_{k}(u_{k}-\mu_{u})^{2}p(u_{k})=\frac{n_{u}}{n_{u}-1}\sum_{k}(u_{k}-\tilde{\mu}_{u})^{2}p(u_{k})=\frac{1}{n_{u}-1}\sum^{n_{u}}_{k=1}(u_{k}-\tilde{\mu}_{u})^{2}\,. (7) These are called, respectively, the sample mean and the sample variance. We use the \tilde{} to distinguish sample quantities. Replacing the true mean with the sample mean in the definition of the sample variance requires adding a corrective term, known as Bessel’s correction, to avoid bias errors (see, among others, Kay, (1993) ). The covariance of the sample is defined similarly for two sequences of samples \mathbf{u},\mathbf{v}\in\mathbb{R}^{n_{u}} \displaystyle\tilde{\mbox{cov}}(\mathbf{u},\mathbf{v})=C_{UV}=\sum_{k}(u_{k}-\mu_{u})(v_{k}-\mu_{v})p(u_{k},v_{k})=\frac{1}{n_{u}-1}(u_{k}-\mu_{u})(v_{k}-\mu_{v})\,. (8) 3.2 Random processes, time series and ergodicity Let us now move from the concepts of random variables to the concept of random processes . This is the natural framework for analyzing stochastic sequences of numbers, i.e. time series (see Shumway and Stoffer, (2011) and Hyndman and Athanasopoulos, (2021) for a comprehensive introduction). A random process is a collection of random variables indexed by time (or space), representing a system’s evolution that changes in a random manner. Focusing on time series analysis, let us u(t) denote our random process; this could be the evolution of one of the velocity components at a specific location. Since a random process is characterized by a distribution of functions, the first and second order statistics are now functions: these are the mean and autocovariance functions : \displaystyle\mu_{u}(t) \displaystyle=\mathbb{E}_{\sim u(t)}\{u(t)\}=\int^{\infty}_{-\infty}uf_{u(t)}(u)du (9) \displaystyle\text{cov}_{u,u}(t_{1},t_{2}) \displaystyle=\mathbb{E}_{\sim u(t)}\{(u(t_{1})-\mu_{u}(t_{1}))(u(t_{2})-\mu_{u}(t_{2}))\} (10) \displaystyle=\int^{\infty}_{-\infty}\int^{\infty}_{-\infty}(u(t_{1})-\mu_{u}(t_{1}))(u(t_{2})-\mu_{u}(t_{2}))f_{u(t_{1}),u(t_{1})}(t_{1},t_{2}))du(t_{1})du(t_{2})\,, having introduced the pdf of the process at a given time, f_{u}(t) , and the joint probability density function of the random variables at times t_{1} and t_{2} f_{u(t_{1}),u(t_{1})}(t_{1},t_{2})) . Note that the autocovariance function is nothing more than the covariance between the random variables obtained by sampling the random process at two different times. This is why this is also often referred to as a “two-point statistics”. The variance at a given time is defined as \sigma^{2}_{u}(t)=\text{cov}_{u}(t,t) . Hence, the auto-correlation function can be obtained by normalizing the auto-covariance as R_{u,u}(t_{1},t_{2})=\frac{\text{cov}_{u}(t_{1},t_{2})}{{\sigma_{u}(t_{1})\,\sigma_{u}(t_{2})}}\,. (11) Similarly, one can compare the degree of similarity between two different random processes. This gives the cross-covariance function between two random processes u(t) and v(t) : \displaystyle\text{cov}_{u,v}(t_{1},t_{2}) \displaystyle=\mathbb{E}_{\sim u(t)}\{(u(t_{1})-\mu_{u}(t_{1}))(v(t_{2})-\mu_{v}(t_{2}))\} (12) \displaystyle=\int^{\infty}_{-\infty}\int^{\infty}_{-\infty}(u(t_{1})-\mu_{u}(t_{1}))(v(t_{2})-\mu_{v}(t_{2}))f_{u(t_{1}),v(t_{1})}(t_{1},t_{2}))du(t_{1})dv(t_{2})\,, and the normalization gives the cross-correlation (the engine of PIV analysis!): R_{u,v}(t_{1},t_{2})=\frac{\text{cov}_{u,v}(t_{1},t_{2})}{\sigma_{u}(t_{1})\sigma_{v}(t_{2})} (13) Note that the expectation operator underpinning all these definitions is an ensemble operator: it averages over all possible realizations of the process at a given time . As with random variables, we cannot work with these integrals directly because we cannot access the necessary probability density functions in practice. Instead, we rely on samples. While samples of a random variable can be collected in a vector, samples of a random process can be organized into a matrix, which we refer to as a snapshot matrix . Let us denote by \mathbf{U}\in\mathbb{R}^{n_{t}\times n_{r}} the snapshot matrix, where each column contains one of the n_{r} realizations, each having n_{t} samples in time: \mathbf{U}=\begin{bmatrix}u(\mathbf{t}_{1},1)&u(\mathbf{t}_{1},2)&\cdots&u(\mathbf{t}_{n_{t}},n_{r})\\ \vdots&\vdots&\cdots&\cdots\\ u(\mathbf{t}_{n_{t}},1)&u(\mathbf{t}_{n_{t}},2)&\cdots&u(\mathbf{t}_{n_{t}},n_{r})\end{bmatrix}\in\mathbb{R}^{n_{t}\times n_{r}} We assume that these samples have been collected on a time list \mathbf{t}=[t_{1},t_{2},\dots,t_{n_{t}}] , which is not necessarily uniformly sampled (that is, {\Delta t}_{i}=\mathbf{t}_{i+1}-\mathbf{t}_{i} is a vector of time differences). The sample mean of the process along these timestamps, denoted as \overline{u}(\mathbf{t}) , is a vector representing the mean over the rows of \mathbf{U} . The sampled auto-covariance function is a matrix that collects the sample covariances across the available samples. This can be conveniently computed via matrix multiplication as: \bm{C}_{UU}(\mathbf{t_{k}},\mathbf{t_{j}})=\frac{1}{n_{r}-1}\mathbf{U}^{\prime}\,\mathbf{U}^{\prime T}\,\in\mathbb{R}^{n_{t}\times n_{t}}\,, (14) where \mathbf{U}^{\prime}=\mathbf{U}-\mu_{U} denotes the mean-centered snapshot matrix, i.e., having to remove the row-wise average to the matrix \mathbf{U} . Note that the entry k,j of this matrix is the inner product of the k -th and the j -th rows of \mathbf{U} (hence all realizations collected at time \mathbf{t}_{k} and \mathbf{t}_{j} ) respectively. The fact that these are not equally spaced does not influence the computation; it only limits the pair of times that can be chosen. The diagonal of this matrix gives the sample variance \tilde{\sigma}^{2}_{u}(\mathbf{t}) at the available time stamps. Normalizing ( 14 ) gives the autocorrelation matrix. Writing \bm{\Sigma}^{2}(t)=\text{diag}(\bm{C}_{UU}) , this can also be computed via matrix multiplication as : \bm{R}_{UU}(\mathbf{t_{k}},\mathbf{t_{j}})=\bm{\Sigma}^{-2}(t)\bm{C}_{UU}(\mathbf{t_{k}},\mathbf{t_{j}}) (15) The expressions to compute sample cross-covariance and sample cross-correlation of an ensemble are analogous and left as an exercise. Two observations are now important: (1) these sample quantities are only available at the time instants \mathbf{t} and (2) while uniform sampling is not required, the sampling locations must remain consistent across all samples of the process. This is the main challenge in tracking velocimetry, where samples are available in different location. A special class of processes are stationary processes . A process is stationary in a strict-sense ( strong stationarity) if the probability density function is constant over time, that is f_{u(t)}(u)=f_{u}(u) , hence df_{u}/dt=0 . A process is stationary in a wide sense ( weak stationarity) if the mean, variance, and autocovariance are constant over time (that is, first and second-order statistics). This means that all two point statistics linking the random variables at time t_{1} and t_{2} only depend on the time difference \tau=t_{2}-t_{1} . Thus we have \displaystyle\mu_{u} \displaystyle=\mathbb{E}_{\sim u(t)}\{u(t)\}=\int^{\infty}_{-\infty}uf_{u}(u)du=\text{const} (16) \displaystyle\text{cov}_{u}(\tau) \displaystyle=\mathbb{E}_{\sim u(t)}\{(u(t)-\mu_{u})(u(t-\tau)-\mu_{u})\} (17) \displaystyle=\int^{\infty}_{-\infty}u(t)u(t-\tau)f_{u(\tau)}(\tau))du\,. Since these notes are solely concerned with first and second-order statistics, the distinction between weak and strong stationarity is irrelevant. The last important concept we review in this section is that of ergodicity . A process is ergodic if ensemble statistics can be replaced by temporal statistics on any particular realization 2 2 2 It is worth noticing that a process might be ergodic for a given statistic but not with others (e.g. a process might be ergodic in the mean but not in the autocorrelation). We will not dig deeper (see Kay, (1993) ; Oppenheim and Verghese, (2017) for more). . This means that the expectation over samples become (see also Lumley, (1970) ; Oppenheim and Verghese, (2017) ; Bruun, (1995) ): \mathbb{E}_{\sim u}\{u(t)\}=\int_{-\infty}^{\infty}u(t)f_{u}(u)du=\frac{1}{T}\int^{T}_{0}u(t)dt=\mathbb{E}_{\sim T}\{u(t)\}\,, (18) where T is the observation time. The underlying idea is that all possible states allowed by the underlying pdf f_{u} are visited for a sufficiently long time series. The proportion of time spent at each state over the observation time T can be seen as the probability of that value occurring in the ensemble. Hence, the change of variables \frac{dt}{T}\approx f_{u}(u)du\quad\text{as}\quad T\rightarrow\infty\, (19) implied in the equality. For a stationary and ergodic process, the autocovariance and the autocorrelation read: C_{UU}(\tau)=\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}(u(t)-\mu_{u})(u(t+\tau)-\mu_{u})\,dt\quad\text{and}\quad R_{UU}(\tau)=\frac{C_{UU}(\tau)}{\sigma^{2}_{u}} (20) where C_{UU}(0)=\sigma^{2}_{u} is the variance of the process. Similarly, the cross-covariance and the cross-correlation between two processes u(t) and v(t) are C_{UV}(\tau)=\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}(u(t)-\mu_{u})(v(t+\tau)-\mu_{v})\,dt\quad\text{and}\quad R_{UV}(\tau)=\frac{C_{UV}(\tau)}{\sigma_{u}\sigma_{v}}\,, (21) with \sigma_{u},\sigma_{v} the standard deviations of the two processes. The statistics of samples can be built by approximating integrals in time. For example, the sample cross-covariance in time between two sample sequences \mathbf{u}(\mathbf{t}) and \mathbf{v}(\bm{t}) , with \mathbf{t}\in\mathbb{R}^{n_{t}} the vector collecting the time instances where the data is available and \Delta\mathbf{t_{k}}=(\mathbf{t_{k+1}}+\mathbf{t_{k}})/2 the time interval representative for each sample becomes: \mbox{cov}_{T}(\mathbf{u},\mathbf{v})=\sum^{n_{t}}_{k=1}({u}(\mathbf{t}_{k})-\mu_{U})({v}(\mathbf{t}_{k})-\mu_{V})\frac{\Delta\mathbf{t_{k}}}{T}\,. (22) Note that, in case of equally sampled time series, with \Delta\mathbf{t_{k}}=\Delta t , one has \Delta t/T=1/n_{t} . The computation of two-point statistics in practice is conceptually more challenging than their ensemble counterparts. Consider a sample \mathbf{u}(\mathbf{t})\in\mathbb{R}^{n_{u}} . To compute the autocovariance at a given lag \tau_{l}=\mathbf{t}_{l+1}-\mathbf{t}_{l-l} , we need to gather many pairs (\mathbf{u}(\mathbf{t}_{j}),\mathbf{u}(\mathbf{t}_{j-l})) and compute the time average of their product. However, this becomes problematic if the timestamps at which the samples are collected do not provide enough such pairs. Binning the possible phase lags could mitigate the problem, but we do not take that route here. Conversely, if the samples are collected at uniform intervals, a sequence of n_{u} entries offers, in principle, 2n_{u}-1 possible lags, and for each lag, at least in principle, n_{u} pairs. We say ‘‘in principle’’, because we here hit the limits of finite duration of the samples. Let us illustrate this limit in the computation of sample auto-covariance 3 3 3 Note that the distinction between covariances and correlations is not universally accepted. Here, we stick to a statistical definition of correlation as normalized covariance, but in numpy/scipy, for example, the function correlate does not normalize the output and thus provides a covariance rather than a correlation. for a sample with uniform sampling \mathbf{t}_{k}=k\Delta t : \text{cov}_{u,u}(\tau_{l})=C_{UU}(\tau_{l})=\mathbb{E}_{\sim t_{i}}\{u^{\prime}(t_{i})u^{\prime}(t_{i}+\tau_{l})\}=\frac{1}{n_{t}}\sum^{n_{t}}_{k=0}u^{\prime}(\mathbf{t_{k}})u^{\prime}(\mathbf{t_{k}}+\bm{\tau}_{l})\,. (23) having used the notation u^{\prime}(\mathbf{t}_{k})=u(\mathbf{t}_{k})-\mu_{U} . In a matrix formalism, this operation could be written as \mathbf{C}_{UU,T}=\frac{1}{n_{t}}\text{circ}(\mathbf{u}^{\prime})^{T}\text{circ}(\mathbf{u}^{\prime}) (24) where circ is the cyclic operator that builds a Toeplitz matrix out of n_{t} lags of a signal, hence: \text{circ}(\mathbf{u}^{\prime})=\begin{bmatrix}u(\mathbf{t}_{1},)&u(\mathbf{t}_{2})&\cdots&u(\mathbf{t}_{n_{t}})\\ \vdots&\vdots&\cdots&\cdots\\ u(\mathbf{t}_{n_{t}})&u(\mathbf{t}_{n_{t}+2})&\cdots&u(\mathbf{t}_{2n_{t}})\end{bmatrix}\in\mathbb{R}^{n_{t}\times n_{t}} (25) The problem of what to put on entries \mathbf{t}_{i} when i is larger than n_{t} . A simple example "by hand" might help illustrate where the problem is with ( 23 ) in the case of finite duration signals. The reader is encouraged to grab paper and pen and use ( 23 ) and compute the autocovariance of the time series u(\mathbf{t_{k}})=[1,2,3,4] , forgetting for the moment the mean removal. Consider a Python-like notation, hence u(t_{0})=0 and u(t_{3})=4 . Considering \mathbf{t}_{k}=k\Delta t , we could use an index notation u[k] and write u[0]=0 or u[3]=4 regardless of the \Delta t . Because this signal has n_{t}=4 entries, we have a total of 2n_{t}-1=7 possible lags l . The lags in ( 23 ) take the form \tau_{l}=l\Delta t , and the autocorrelation is thus a vector of length 7 which can also be indexed as \tilde{C}_{UU}(\tau_{l})=\tilde{C}_{UU}[l] with l\in[-3,3] . Figure 3 illustrate the problem of computing \tilde{C}_{UU}[-2] : this is the correlation between the signal u[k] and its copy shifted by 2 entries to the right u[k-2] . Figure 3 : Sketch of the difference between linear and cyclic autocorrelation. The figure shows how these two definitions compute the autocorrelation entry \tilde{R}_{UU}[-2] of the vector u[k]=[1,2,3,4] . The linear correlation pads with zero on both sides while the cyclic correlation assumes periodicity with period n_{t} . While it is clear that the summation will contain 3\times 1+4\times 2 , it is unclear how to deal with the entries that do not have a match in the other signal because of the finite size of the signal. On a practical level, the two classic solutions are zero padding and cyclic padding . The zero-padding leads to the linear correlation : the vectors are padded by zeros in all the entries lacking information. The cyclic padding leads to the cyclic correlation : the vectors are assumed periodic with period n_{t} . We distinguish these operators with and L or a C , and the results for this example is: \tilde{C}_{UUL}[-2]=11/7\quad\tilde{C}_{UUC}[-2]=22/7\,. (26) Note that C_{UUC} is periodic and for both operators we have \tilde{C}_{UU}[l]=\tilde{C}_{UU}[-l] . This is why one often plots only 4 4 4 The reader is encouraged to use ( 23 ) to compute the linear and the circular autocovariance of u[k]=[1,2,3,4] . Focusing on the ‘positive shifts’, one has \tilde{C}_{UUL}=[30,20,11,4]/7 and \tilde{C}_{UUC}=[30,24,22,24]/7 . the last n_{t} vectors of the autocorrelation, corresponding to the lags from l=0 to l=n_{t}-1 . When the time series are ‘short’, \tilde{C}_{UUL} and \tilde{C}_{UUC} differ significantly, and it is essential to clarify which is being used. ‘Short’ here means that relevant lags (giving normalized cross-correlation of \sim 1 ) are present within a time scale that is comparable to the duration of the signal. This is the case of PIV interrogation (see Käufer, (2020) ), in which time is replaced by space, and the discrete-time indices are the shifts in pixels. This also explains the importance of the classic “1/4" rule: the displacement should not exceed a quarter of the window or else the boundary problems becomes too important. The linear (acyclic) operators are unbiased estimators ( Kay,, 1993 ) and provide better statistical convergence to their ensemble counterparts. On the other hand, the cyclic operators are computationally more interesting because the periodic assumption enables a link with the cyclic convolution, which can be computed in the frequency domain using the Fast Fourier Transform (FFT). Briefly, it is easy to note that the cyclic convolution between two vectors x[k] and y[k] differs from the cyclic covariance by the flipping of the second vector (see Hayes, (2011) ). In the frequency domain, this flipping corresponds to the conjugation of the associated Fourier transform. To be more specific, let X(f_{n}) be the Discrete Fourier Transform (DFT) of x[n] , i.e.: X(f_{n})=\mathcal{F}\{x[k]\}=\frac{1}{n_{t}}\sum^{n_{t}}_{k=0}x[k]e^{-2\pi f_{n}k\Delta t}\leftrightarrow x[k]=\mathcal{F}^{-1}\{X[k]\}=\sum^{n_{t}}_{k=0}X(f_{n})e^{2\pi f_{n}k\Delta t} (27) where f_{n}=n\Delta f , with n=0,n_{t}-1 and \Delta f=f_{s}/n_{t} the frequency resolution, with fs=1/\Delta t the sampling frequency. The cyclic cross-correlation between two signals x[k] , y[k] , having DFT X(f_{n}) , Y(f_{n}) can be computed as R_{XY}=\mathcal{F}^{-1}\biggl(\mathcal{F}\{x[k]\}\overline{\mathcal{F}\{y[k]\}}\biggr)=\mathcal{F}^{-1}\bigl(X(f_{n})\overline{Y}(f_{n})\bigr)\,, (28) with the overbar denoting complex conjugation. A Python implementation of the cyclic cross correlation is thus provided by following function: ⬇ 1 def R_UW_C ( u , v ): 2 RUU = np . fft . ifft ( np . abs ( np . fft . fft ( u )* np . fft . fft ( v ))). real 3 c =( RUU / len ( u )-( np . mean ( u )* np . mean ( v )))/( np . std ( u )* np . std ( v )) 4 return c [: len ( u )//2] It is possible to compute the linear operators using a circular extension if an appropriate zero padding is used before the cyclic extension (see Smith, 2007b () ). This is what software packages like Scipy and Numpy do when computing the ’FFT-based’ cross-correlation. The linear cross-correlation using the Scipy ’s function correlate is implemented in the following function: ⬇ 1 from scipy import signal 2 def R_UW_L ( u , v ): 3 # Call to the scipy function correlate: 4 RUU = signal . correlate ( u - np . mean ( u ), v - np . mean ( v ),\ 5 mode = ’same’ , method = ’auto’ )/( len ( u )-1)/( np . std ( u )* np . std ( v )) 6 return RUU [ RUU . size //2:] The entry ‘method’, set by default to ‘auto’, uses the fastest option between direct and FFT-based correlation, depending on the size of the array. These functions are needed for the next exercise. Exercise 1: The statistics of an Ornstein-Uhlenbeck Process Consider the Ornstein-Uhlenbeck process governed by the following stochastic differential equation (see Øksendal, (1998) ; Borodin and Salminen, (2002) ; Pope, (2000) ) dX_{t}=\kappa(\theta-X_{t})dt+\sigma dW_{t} (29) where the subscript t here refers to a time step, \kappa>0 is called rate of reversion and controls how quickly the process reaches its (stationary) long-term behavior, characterized by mean \theta and random fluctuation with standard deviation \sigma . The second term W_{t} is a stochastic term, and it is here taken as a normal Gaussian with zero mean and unitary standard deviation, here written as \mathcal{N}(0,1) . The first term is often referred to as drift , the second as diffusion . This process is closely related to the Langevin equation used in Lagrangian turbulence modeling ( Thomson,, 1987 ; Pope,, 1994 ) , where an equation similar to ( 29 ) models the velocity of a particle as it moves through a turbulent flow. Consider a case with initial condition X_{0}=0 , \kappa=1.2 , \theta=3 and \sigma=0.5 . Consider n_{t}=1001 samples, with a sampling of \Delta t=0.01 s. This corresponds to a physical observation time of T=10s . Plot four realizations to explore the process. The questions are two. (1) compute the ensemble mean, the ensemble standard deviation and the ensemble autocovariance as a function of time using n_{r}=100 and considering the steady state condition. (2) Study the convergence of these statistical quantities at two time steps (say at k=10 and k=700 ) as a function of the number of samples in the ensemble. Solution . Let us begin by creating a function that produces a sample of the process, taking as input the four process parameters \kappa,\theta,\sigma and vector of times \mathbf{t_{k}} . Using a simple loop, the following function does the job (see Python exercise 1): ⬇ 1 import numpy as np 2 # Function definition 3 def U_O_Process ( kappa , theta , sigma , t ): 4 n_T = len ( t ) 5 # Initialize the output 6 y = np . zeros ( n_T ) 7 # Define Drift and Diffusion functions in the process 8 drift = lambda y , t : kappa *( theta - y ) 9 diff = lambda y , t : sigma 10 noise = np . random . normal ( loc =0, scale =1, size = n_T )* np . sqrt ( dt ) 11 # Solve Stochastic Difference Equation 12 for i in range (1, n_T ): 13 y [ i ]= y [ i -1]+ drift ( y [ i -1], i * dt )* dt + diff ( y [ i -1], i * dt )* noise [ i ] 14 return y The following script creates n_{r}=500 realizations and store them in a matrix \bm{U}_{n} of size n_{t}\times n_{r} (see ( 14 )). Then, it plots four randomly chosen samples, namely the numbers r=1,10,22,55 . The four realizations are shown in Figure 4(a) , with the plot axis being customized (see the provided codes). ⬇ 1 import matplotlib . pyplot as plt 2 # Initial and final time 3 t_0 =0; t_end =10 4 # Number of Samples 5 n_t =1001 6 # Process Parameters 7 kappa =1.2; theta =3; sigma =0.5 8 # Create the time scale 9 t = np . linspace ( t_0 , t_end , n_t ); dt = t [2]- t [1] 10 # Collect 500 sample and store in U_N 11 n_r =500; U_N = np . zeros (( n_t , n_r )) 12 for l in range ( n_r ): 13 U_N [:, l ]= U_O_Process ( kappa , theta , sigma , t ) 14 # Plot the results 15 plt . figure (1) 16 plt . plot ( t , U_N [:,1]) 17 plt . plot ( t , U_N [:,10]) 18 plt . plot ( t , U_N [:,22]) 19 plt . plot ( t , U_N [:,55]) The reader should play with the code long enough to realize that this process is characterized by a transient time of the order of 3s within which all signals move from zero to a stationary condition. Having arranged the data into ‘snapshot matrices’, the computation of the time average and the temporal standard deviation can be done in one line each: ⬇ 1 U_Mean = np . mean ( U_N , axis =1) # Ensemble Mean 2 U_STD = np . std ( U_N , axis =1) # Ensemble STD These are essentially ‘row-wise’ statistics. Figure 4(b) shows the time average together with the range \mu_{U}(\mathbf{t_{k}})\pm\sigma_{U}(\mathbf{t_{k}}) . It appears that the standard deviation grows gently until it reaches a constant value after about \mathbf{t_{k}}>4s . Finally, we analyze the ensemble autocorrelation of this random process. The ensemble cross-correlation between the time steps \mathbf{t_{j}} and \mathbf{t_{k}} for the ensembles \mathbf{U}_{n} and \mathbf{W}_{n} of two random variables can be computed with the following function: ⬇ 1 def Ensemble_Autocorr ( U_N , W_N , k , j ): 2 n_r , n_t = np . shape ( U_N ) 3 # Select all realizations at time t_k for U 4 U_N_k = np . expand_dims ( U_N [ k ,:], axis =0); 5 # Select all realizations at time t_kj for W 6 W_N_k = np . expand_dims ( W_N [ j ,:], axis =0) 7 # Note (These are row vectors) 8 # Compute the average products 9 PR = U_N_k . T . dot ( W_N_k ) 10 R_UW = np . mean ( PR )/( np . std ( U_N_k )* np . std ( W_N_k )) 11 return R_UW (a) (b) Figure 4 : Fig (a): Four randomly chosen samples of the Ornstein- Uhlenbeck process; Fig (b): Ensemble average of the process (in blue), along with the \mu_{u}+\sigma and \mu_{u}-\sigma curves. Having added a normalization. Then, the following script computes the autocorrelation of 100 randomly pairs, separated by a lag of 50 samples: ⬇ 1 # Define lag (in number of samples) 2 lag =50 3 # Study the autocorrelation between two points at equal lags 4 N_S =100; R_UW = np . zeros ( N_S ) 5 # Select a 100 random points i (larger than 500) 6 J = np . random . randint (500,800, N_S ); K = J +50 7 for n in range ( N_S ): 8 R_UW [ n ]= Ensemble_Autocorr ( U_N , U_N , J [ n ], K [ n ]) The resulting set of autocorrelations has a mean of 0.545 and a standard deviation of 0.0059 : in other words, it does not matter what the exact pair j,k is, as long as they differ by the same lag (in this case, 50). This is because here we are sampling in time intervals from 500 to 800, and here the process has reached its stationary condition. The reader is encouraged to repeat the exercise at an earlier interval. Finally, we analyze the convergence of the statistics as a function of the number of realizations. The following script computes the mean and the standard deviation at k=10 and k=700 for a number of realizations that goes from 1 to 1000 . ⬇ 1 n_R = np . round ( np . logspace (0.1,3, num =41)) 2 # Prepare the outputs at k=100 3 mu_10 = np . zeros ( len ( n_R )) 4 sigma_10 = np . zeros ( len ( n_R )) 5 # Prepare the outputs at k=700 6 mu_700 = np . zeros ( len ( n_R )) 7 sigma_700 = np . zeros ( len ( n_R )) 8 # Loop over all n_R’s. 9 for n in range ( len ( n_R )): 10 # show progress 11 print ( ’Computing n=’ + str ( n )+ ’ of ’ + str ( len ( n_R ))) 12 n_r = int ( n_R [ n ]) # Define the number of ensembles 13 U_N = np . zeros (( n_t , n_r )) # Initialize the ensemble set 14 for l in range ( n_r ): # Fill the Ensemble Matrix 15 U_N [:, l ]= U_O_Process ( kappa , theta , sigma , t ) 16 # Compute the mean and the std’s 17 mu_10 [ n ]= np . mean ( U_N [10,:]) # Ensemble Mean 18 sigma_10 [ n ]= np . std ( U_N [10,:]) # Ensemble STD 19 mu_700 [ n ]= np . mean ( U_N [700,:]) # Ensemble Mean 20 sigma_700 [ n ]= np . std ( U_N [700,:]) # Ensemble STD By analyzing the vector n_R in the script, the reader should note that some of the entries with a small number of samples are taken multiple times to show the variability in the prediction. The results are shown in Figure 5 . It can be shown that the convergence of both the mean and the standard deviation is \propto\sqrt{n_{r}} , but the convergence of the mean is \propto\sigma_{U} while the convergence of the standard deviation is \propto\sigma^{2}_{U} (see also Ianiro, (2020) ): this explains why a larger number of samples is needed to converge the second-order statistics, and why the convergence at k=100 is slower. The reader is encouraged to explore the provided scripts further, to observe this result in action at different points. (a) (b) Figure 5 : Fig (a): mean velocity convergence; Fig (b): standard deviation convergence. 3.3 Power spectral densities and cross-coherency To illustrate the use of second-order statistics in spectral analysis, this section moves to the frequency representation of stochastic signals. This is slightly more involved than the frequency representation of deterministic signals. In a continuous domain, a stochastic signal is seldom square-integrable, and it thus seldom admits an ordinary Fourier transform. In the signal processing terminology, the definition of an appropriate frequency domain requires shifting the treatment from the notion of energy to the notion of power 5 5 5 Given an infinite duration stochastic signal x[k] , indexed by the integers k , the energy \mathcal{E} and the power \mathcal{P} are defined as follows: \mathcal{E}\{x[k]\}=\lim_{n_{t}\to\infty}\sum^{n_{t}}_{k=0}|x[k]|^{2}\quad\quad\mathcal{P}\{x[k]\}=\lim_{n_{t}\to\infty}\frac{1}{2n_{t}-1}\sum^{n_{t}}_{k=0}|x[k]|^{2}\,. (30) : stochastic signals have generally infinite energy but finite power ( Oppenheim and Verghese,, 2017 ; Hayes,, 2011 ) . Therefore, we shall not focus on the Fourier transform of a signal but on the Fourier transform of its autocorrelation (which is square integrable). The following transform exists for all signals of interest: \mathbb{E}\{u^{2}_{n}(\mathbf{t_{k}})\}={R}_{UU}(0)=\frac{1}{2\pi}\int^{\infty}_{-\infty}S_{UU}(\omega)d\omega\,, (31) where \omega=2\pi f is the pulsation, f is the frequency and S_{UU} is the continuous Fourier transform of the autocorrelation function. This is known as power spectral density . Note that this is real and even in \omega because the time autocorrelation R_{UU} is symmetric ( R_{UU}(\tau)=R_{UU}(-\tau) ). The Fourier pairs of interest are thus S_{UU}(\omega)=\int^{\infty}_{-\infty}{R}_{UU}(\tau)e^{-\mathrm{j}\omega\tau}d\tau\quad\mbox{and}\quad{R}_{UU}(\tau)=\frac{1}{2\pi}\int^{\infty}_{-\infty}S_{UU}(\omega)e^{\mathrm{j}\omega\tau}d\omega\,. (32) These are also known as Wiener-Khinchin relations . Similarly, the cross-spectral density and the cross-correlation are Fourier pairs: S_{UV}(\omega)=\int^{\infty}_{-\infty}{R}_{UV}(\tau)e^{-\mathrm{j}\omega\tau}d\tau\quad\mbox{and}\quad{R}_{UV}(\tau)=\frac{1}{2\pi}\int^{\infty}_{-\infty}S_{UV}(\omega)e^{\mathrm{j}\omega\tau}d\omega\,. (33) The notion of power spectral density is important because it allows us to give a spectral representation to stochastic signals and thus to generalize the theory of linear time-invariant (LTI) systems (see Mendez, (2020) ). This theory brings powerful and simple tools for system identification, filtering and forecasting (see Oppenheim and Verghese, (2017) ; Smith, 2007a () for more). Concretely, we are interested in the notion of coherency in relation to the frequency content of signals. We might want to infer, for example, how much the frequency content of two time series are linked, at least within a certain range of frequencies. This could help identify for example, if a given pattern is “traveling” between two probes (e.g. Probes P2 and P3 in the test case in section 2.1 ). We first need some definitions. Let us assume that a stochastic signal x[k]\in\mathbb{R}^{n_{t}} is the input of a linear and deterministic system which responds with a second stochastic signal y[k]\in\mathbb{R}^{n_{t}} . Uncorrelated noise might be added to the output of this system and we only see the resulting y_{n}[k]=y[k]+N[k] . If the power of the noise is too large, we might not be able to recover any reasonable estimate of y[k] and we will say that there is a poor level of coherency between x[k] and y_{n}[k] . Conversely, we might be able to identify an approximation of the underlying linear linking input and output. This link is here to be analyzed frequency by frequency. Considering finite duration signals, the convergence problems of the Fourier representation are less stringent, and we can waive some of the formalism required for the continuous world: under the assumption of circular extension of the signals 6 6 6 In this section we will only consider cyclic padding. The zero-padding requires some little extra care which is not essential for this lecture (see Smith, 2007b () ).) , every digital signal has a Discrete Fourier Transform (DFT). Let X(f_{n}) and Y(f_{n}) denote the DFT of the input x[k] and the output y[k] (see ( 27 ) for the definitions). If a linear time invariant system links y[k] to x[k] , the output can be computed via convolution of the input with the system’s impulse response. From the convolution theorem, we know that in frequency domain this is a multiplication with the transfer function of the system H(f_{n}) , i.e. the DFT of the impulse response. We thus have: y[k]=\sum^{n_{t}-1}_{m=0}x[k]\,h[k-m]\,\,\longleftrightarrow\,\,Y(f_{n})=H(f_{n})\,X(f_{n})\,. (34) We now introduce the discrete equivalent of ( 33 ). It is possible to show (see Oppenheim et al., (1996) and eq. ( 28 )) that the power spectral density of the input is S_{XX}(f_{n})=\sum^{n_{t}-1}_{k=0}R_{XX}[k]e^{-2\pi f_{n}k\Delta t}=\mathcal{F}\{\mathcal{F}^{-1}\{X(f_{n})\overline{X}(f_{n})\}\}=X(f_{n})\overline{X}(f_{n})\,. (35) Similarly, the power spectral density of the output is S_{YY}(f_{n})=Y(f_{n})\overline{Y}(f_{n}) and we can also define cross-spectral density as S_{YX}=X(f_{n})\overline{Y}(f_{n}) . We can now craft a spectral function which measures how well the output spectrum correlates with the input spectrum. This is the coherence function : \widehat{C}_{YX}(f_{n})=\frac{|S_{YX}(f_{n})|^{2}}{S_{XX}(f_{n})S_{YY}(f_{n})}\,\,. (36) At frequencies for which Y(f_{n})=H(f_{n})X(f_{n}) (i.e., for which an LTI system could model the input-output relation), one has S_{YX}(f_{n})=Y(f_{n})\overline{X}(f_{n})=H(f_{n})X(f_{n})\overline{X}(f_{n})\rightarrow S_{YX}(f_{n})=H(f_{n})S_{XX}(f_{n})\,. Moreover, noticing that S_{YY}(f_{n})=|H(f_{n})|^{2}S_{XX}(f_{n}) , we recover \widehat{C}_{YX}(f_{n})=1 . At frequencies for which no LTI system can link the two spectra, coherence is zero. The function is undefined at f_{n} ’s for which either S_{XX} or S_{YY} is zero. To conclude this section, it is worth noticing that working with one single spectrum for both input and output makes little sense: an ensemble of time series leads to an ensemble of spectra. The classic approach to evaluating a stochastic signal’s frequency representation involves averaging, which could be either in the ensemble or time domains. Under the assumption of ergodicity, the second is usually preferred, and the result is the well-known Welch’s method ( Welch,, 1967 ) or periodogram method . The computation is performed by dividing the signal into successive (and overlapping) blocks, computing the DFT, and then averaging the results. In practice, a smoothing window w[k] multiplies the signal in the time domain to deal with the problems arising from the (usually violated) periodicity assumption. For S_{XX}(f_{n}) , for instance, we have: S_{XX}(f_{n})=\frac{1}{n_{L}}\sum^{n_{L}-1}_{m=1}|XW_{m}(f_{n})|^{2}\quad\mbox{with}\quad XW_{m}(f_{n})=\mathcal{F}\{x[k]\,w[k]\} (37) Here XW_{m}(f_{n}) is the DFT of the windowed block x_{m}[k]w[k] , with x_{m}[k]=x[k+mn_{W}] , w_{m}[k] the window function designed to gracefully taper to zero at both endpoints of the block, and k=0,1,\dots n_{w}-1 with n_{L} denoting the number of blocks. Because the multiplication in the time domain is a convolution in the frequency domain, the method is essentially a smoothing operation on the signal’s spectrum; this is why the method is sometimes also called ‘smoothed spectrum’ and implemented in the frequency domain. The Python ’s scipy package offers robust functions to compute both the power spectral densities and the spectral coherence. We use both in the next exercise. Exercise 2: Power Spectral Densities and Coherency Consider the time series sampled in probes P2 and P3 (see Figure 1 ) a. (1) Study the covariance and correlation between the two signals in the first t<1 s of observation, i.e. when the flow is in its first stationary condition. Then, (2) analyze the power spectral densities, the cross spectral densities and the cross-coherency between the two signals. Solution . The solution to this exercise is provided in the file Exercise_2.py. Figure 6 shows a scatter plot of the stream-wise component in P3 versus P2. The correlation coefficient between the two time series is \rho_{1,2}=0.58 , which is rather high for a turbulent flow. Figure 7 provides the cross spectral density S_{XY}(f_{n}) (Figure a) and the cross coherency C_{X,Y}(f_{n}) (figure b) between the two signals. Figure 6 : Scatter plot of the streamwise velocity components in probe P3 versus probe P2 (see Figure 1 for the location. A non-negligible correlation is visible. A strong correlation between the signals is evident near the vortex shedding frequency, which in this signal segment is approximately 440 Hz. The cross-coherency level is nearly unity, indicating that the two signals could be related within this frequency range through a linear time-invariant (LTI) system. This is due to the global nature of the oscillation mechanism in the vortex shedding, which synchronizes large portions of the flow ( in this case, the entire wake) in a coherent oscillatory behavior. (a) (b) Figure 7 : Fig (a): Cross spectral density; Fig (b): Cross coherency between signal P1 and P2 4 The statistical treatment of turbulence When analyzing turbulent flow statistics, the time series analysis tools introduced in Section 3.2 must be extended in two ways: (1) the random process is a velocity signal, which is a vector quantity, and (2) the random process depends not only on time but also on space. The following subsections discuss these two extensions separately. We stress that the statistical treatment of turbulence is a vast subject, far beyond the scope of this lecture notes. The reader is referred to Pope, (2000) ; Tennekes, (1972) ; Davidson, (2004) ; Mcdonough, (2004) for a comprehensive introduction to the topic and to Saarenrinne and Piirto, (2000) ; Lavoie et al., (2007) ; Segalini et al., (2014) ; Scharnowski et al., (2018) ; Ayegba and Edomwonyi-Otu, (2020) ; Wang et al., (2021) for a discussion on the impact of measurement resolution on the main turbulence variables. This section recalls the definitions that are required to solve the exercises provided. 4.1 Local statistics of velocity components Let \bm{u}(\bm{x},t) denote the velocity field, having components \bm{u}:=(u,v,w) at each location \bm{x}:=(x,y,z) and time t . Let us introduce a more compact notation \mathbb{E}_{E}\{a\}=\langle a\rangle_{E} to denote the expectation operator on the set of samples E . In the classic Reynolds decomposition of turbulent flows, the velocity field is decomposed into the sum of an ensemble average and a fluctuating part, that is \bm{u}(\bm{x},t)=\langle\bm{u}\rangle_{E}(\bm{x},t)+\bm{u}^{\prime}(\bm{x},t)\,, (38) where \langle\mathbf{u}\rangle=(\langle u\rangle,\langle v\rangle,\langle w\rangle)^{T} is ensemble average field and \mathbf{u}^{\prime}=(u^{\prime},v^{\prime},w^{\prime})^{T} is the fluctuating field. The averaging in this formulation is an ensemble averaging and should not be confused with the time averaging introduced in the following for stationary data. While this ensemble averaging acts as a smoother of the small-scale fluctuations, this should not be confused with the frequency-based filtering employed in the Large Eddy Simulation (LES) formalism. An example might help clarify this distinction. Consider the case of a transient flow in a channel, with the flow ramping up from zero to a fully established regime within a time interval T . That is, let us assume that the flow rate follows the same time evolution as the stochastic process analyzed in Exercise 1. A filtering in the time domain, as in the LES formalism, would impose a cut-off frequency that removes components above a certain value, which may smooth out some of the frequencies associated with the ramp-up phase. In contrast, under an ensemble averaging approach, we would run the experiment multiple times and take averages over all realizations, as done in Exercise 1. Such ensemble averaging produces time-dependent "mean flows" which are described by the Unsteady Reynolds-Averaged Navier–Stokes (URANS) formalism. The main conceptual difficulty here is that this operation has a complex relationship with the frequency content remaining after ensemble averaging: the individual realizations could all exhibit strong gradients (thus high-frequency components) in certain regions, which would be preserved in the average. Therefore, the ensemble-averaged flow may still be time-dependent and retain high-frequency content. The key takeaway is that URANS, in general, is not equivalent to time averaging and does not inherently smooth out high-frequency fluctuations. Particularly interesting, in an ensemble averaging formalism, are the second order statistics linking fluctuations along the different components. At each location \bm{x} and for each time t , the covariance of the velocity component is defined as \bm{R}(\bm{x},t)=\begin{pmatrix}\langle u^{\prime 2}\rangle&\langle u^{\prime}v^{\prime}\rangle&\langle u^{\prime}w^{\prime}\rangle\\ \langle v^{\prime}u^{\prime}\rangle&\langle v^{\prime 2}\rangle&\langle v^{\prime}w^{\prime}\rangle\\ \langle w^{\prime}u^{\prime}\rangle&\langle w^{\prime}v^{\prime}\rangle&\langle w^{\prime 2}\rangle\end{pmatrix}\,, (39) having omitted the subscript E in the expectations. This matrix is known as Reynolds Stress Tensor, which arises when introducing the decomposition ( 38 ) into the Navier Stokes Equations and then ensemble averaging. The result from these operations are the URANS equations; for an incompressible flow with constant properties and negligible volume forces reads: \rho(\partial_{t}\langle\bm{u}\rangle+\langle\bm{u}\rangle\nabla\langle\bm{u}\rangle)=\nabla\langle p\rangle+\mu\nabla^{2}\langle\bm{u}\rangle-\rho\nabla\cdot\bm{R}\,, (40) where \rho and \mu are the density and dynamic viscosity, p is the pressure, and the last term models the effects of turbulence on the mean flow \langle\bm{u}\rangle , and its mathematically equivalent to additional stress. The form of the Reynolds stress gives, therefore, information about the intensity of the turbulent fluctuations and the direction across which turbulence is most prominently increasing the mean flow stresses. Turbulence intensity is usually expressed in terms of root mean square of the fluctuation u_{rms} , computed from the turbulent kinetic energy \kappa=\{\bm{R}\}/2 , with \{\} denoting the trace of a matrix, and a reference velocity U : \mbox{TI}=\frac{u_{rms}}{U}=\frac{1}{U}\sqrt{\frac{2}{3}\kappa}\,. (41) The Reynolds stress can be used to characterize various states of turbulence . The simplest state is that of isotropic turbulence , in which turbulent fluctuations are directionally independent and have no preferential orientation. In this case, the Reynolds stress is diagonal, with all components equal u_{rms}=\langle u^{\prime 2}\rangle=\langle v^{\prime 2}\rangle=\langle w^{\prime 2}\rangle . A simple way to characterize the level of anisotropy in the flow is the anisotropy stress tensor \bm{A}=\frac{1}{2\kappa}\bm{R}-\frac{1}{3}\bm{I}\,, (42) where \bm{I} is the identity matrix. This is identically zero in the case of isotropic turbulence, hence its Frobenious norm ||\bm{A}||_{F}=\sqrt{\sum_{i}\sum_{j}\bm{A}_{i,j}} gives a first quantitative measure of the level of anisotropy. More insights on the preferential directionality of turbulence fluctuations can be obtained by the eigenvalue decomposition of the anisotropic tensor matrix ( Emory and Iaccarino,, 2014 ) . These can be used to compute two coordinates in a 2D map, known as an invariant map, from which various states of turbulence can be visualized. A popular map is the so-called Lumley triangle ( Lumley,, 1978 ) , which uses the second and third principal components of turbulence anisotropy; the coordinates in this plane are: \mbox{II}=\lambda^{2}_{1}+\lambda_{1}\lambda_{2}+\lambda^{2}_{2}\quad\mbox{and}\quad\mbox{III}=-\lambda_{1}\lambda_{2}(\lambda_{1}+\lambda_{2})\,, (43) with \lambda_{1}>\lambda_{2}>\lambda_{3} the ordered eigenvalues of the anisotropy tensor. Three limiting states can be found in this plane (see Emory and Iaccarino, (2014) ): 1. One-component turbulence : Fluctuations only exists along one direction. This occurs if \lambda_{i}=[2/3,-1/3,-1/3]^{T} . This condition is represented by \bm{x}_{1C} in all invariant maps. 2. Axisymmetric two-components : Fluctuations have two leading directions with equal magnitude. This occurs if \lambda_{i}=[1/6,1/6,-1/3]^{T} . This condition is represented by \bm{x}_{2C} in all invariant maps. 3. Isotropic turbulence : Fluctuations are equally relevant in all directions (spherical turbulence). As previously mentioned, the anisotropic tensor is identically zero; hence, one has \lambda_{i}=[0,0,0]^{T} . This condition is represented by \bm{x}_{3C} in all invariant maps. These points can be joined to identify the boundaries of the invariant map, which also correspond to different physical behaviors. In particular: 1. Joining \bm{x}_{1C} and \bm{x}_{3C} : this occurs when 0<\lambda_{1}<1/3 and -1/6<\lambda_{2}=\lambda_{3}<0 and corresponds to an axisymmetric expansion. 2. Joining \bm{x}_{2C} and \bm{x}_{3C} : this occurs when -1/3<\lambda_{1}<0 and 0<\lambda_{2}=\lambda_{3}<1/6 and corresponds to an axisymmetric contraction. 3. Joining \bm{x}_{1C} and \bm{x}_{2C} : this occurs when \lambda_{1}+\lambda_{3}=1/3 and \lambda_{2}=-1/3 and corresponds to the line of two components turbulence. Figure 8 plots the Lumley triangle, to which we return in the next exercise. Figure 8 : Lumley triangle and invariant map construction The properties of the Reynolds stress tensor can also be used to define realizability conditions for a turbulence model to be physical Gerolymos and Vallet, (2016) . The reader is referred to Stiperski and Calaf, (2018) ; Oberlack and Guenther, (2002) for more details. Finally, we close with the special case in which the flow is statistically stationary and ergodic . In this case, as discussed in section 3.2 , ensemble averaging can be replaced by time averaging over a sufficiently long time series. Both the mean flow and the Reynolds stress are no longer a function of time, and all two points’ statistics in time are solely functions of the time shift \tau (equations ( 16 )). This also implies that the autocorrelation of all velocity components at any given location vanishes at \tau\rightarrow\infty . For a vector-valued time series, it is convenient to define the time autocorrelation as R_{\bm{u}}(\bm{x},\tau)=\int^{\infty}_{0}\bm{u}^{\prime}(\bm{x},\tau)^{T}\bm{u}^{\prime}(\bm{x},t+\tau)d\tau\,, (44) with T denoting transposition. One thus has that R_{\bm{u}}(\bm{x},0)=||\bm{u}^{\prime}(x)||_{2}^{2} , with ||\bm{a}||_{2} the l_{2} norm of a vector \bm{a} . Since R_{\bm{u}}\rightarrow 0 for a stationary flow, it is possible to define an integral time scale as (see also Oliveira et al., (2007) ) as \Theta=\int^{\infty}_{0}\frac{R_{\bm{u}}(\bm{x},\tau)}{R_{\bm{u}}(\bm{x},0)}d\tau\,. (45) It is worth stressing that this integral converges (and the definition makes sense) only if the autocorrelation function tends to zero. In practice, one should acquire a time series sufficiently long to see the autocorrelation becoming negligible. The integral time scale measures the time after which the random process becomes uncorrelated with itself or, in a more pictorial interpretation, the time within which the variable ‘remembers’ its history. 4.2 Global statistics in space and time In Section 3.2 , the second-order statistics were aimed at linking two possible outcomes of the process at different times, with the outcome at each time being a random variable. In Section 4.1 , these were used to link the components of the vector field at a given location and a given time; again, each of these components is a random variable. In this section, we aim to use second-order statistics to study similarities in space and time. In space analysis, we seek to link the time series of the velocity fields sampled at different locations. We thus define the spatial covariance function as C_{s}(\bm{x},\bm{x}^{\prime})=\int_{E}\bm{u}^{\prime}(\bm{x},t)^{T}\bm{u}^{\prime}(\bm{x}^{\prime},t)f_{\bm{u}(x),\bm{u}(x^{\prime})}d\bm{u}\,. (46) This definition generalizes ( 12 ): the integral is carried out over all the possible set of time series occurring at location \bm{x} and \bm{x}^{\prime} and f_{\bm{u}(x),\bm{u}(x^{\prime})} is the (unknown) joint probability distribution. The inner product is used to obtain a scalar function from the vector-valued samples. For a stationary and ergodic process, leveraging ( 19 ), one can exchange ensemble averaging with time averaging and obtain the spatial covariance function : C_{s}(\bm{x},\bm{x}^{\prime})=\frac{1}{T}\int_{T}\bm{u}^{\prime}(\bm{x},t)^{T}\bm{u}^{\prime}(\bm{x}^{\prime},t)dt\,. (47) A special property of turbulent flows is that of homogeneity : in homogeneous turbulence, the spatial covariance function solely depends on the distance between the two points considered, hence C_{s}(\bm{r})=C_{s}(\bm{x},\bm{x}+\bm{r}) . In homogeneous and isotropic turbulence, it can be shown that it solely depends on the magnitude ||\bm{r}|| of the shift and not its direction. The autocovariance, in this case, can be used to define an integral length scale similarly to the integral time scale introduced earlier. A vector-based variant of the spatial correlation matrix can be defined as follows \bm{C}(\bm{x},\bm{x^{\prime}})=\begin{bmatrix}\mbox{cov}_{T}(u(\bm{x}),u(\bm{x^{\prime}}))&\mbox{cov}_{T}(u(\bm{x}),v(\bm{x^{\prime}}))&\mbox{cov}_{T}(u(\bm{x}),w(\bm{x^{\prime}}))\\ \mbox{cov}_{T}(v(\bm{x}),u(\bm{x^{\prime}}))&\mbox{cov}_{T}(v(\bm{x}),v(\bm{x^{\prime}}))&\mbox{cov}_{T}(v(\bm{x}),w(\bm{x^{\prime}}))\\ \mbox{cov}_{T}(w(\bm{x}),u(\bm{x^{\prime}}))&\mbox{cov}_{T}(w(\bm{x}),v(\bm{x^{\prime}}))&\mbox{cov}_{T}(w(\bm{x}),w(\bm{x^{\prime}}))\,.\end{bmatrix} (48) The reader should remark the similarity with the Reynolds stress tensor defined in Section 4.1 : indeed, the only difference is that this is now a function of two locations while the Reynolds stress tensor is defined at one location. Both are second order statistics, but \bm{R}(\bm{x}_{i}) is a covariance matrix comparing the velocity components at location \bm{x}_{i} while \bm{C}(\bm{x},\bm{x^{\prime}}) is a covariance comparing velocity components at locations \bm{x} and \bm{x}^{\prime} . In the time analysis, we seek to link the velocity fields sampled at two different times. We thus define the temporal covariance function as K(t,t^{\prime})=\int_{E}\bm{u}^{\prime}(\bm{x},t)^{T}\bm{u}^{\prime}(\bm{x},t^{\prime})f_{\bm{u}(t),\bm{u}(t^{\prime})}d\bm{u}\,. (49) Again, this ensemble operator is a generalization of ( 12 ). This time, we are considering the set of all possible fields that could occur in two time steps. As for the previous operator, we might invoke spatial ergodicity: assuming that the domain is sufficiently large to display all the possible outcomes of the time series involved, we could replace ensemble averaging with spatial averaging to obtain: K(t,t^{\prime})=\frac{1}{|\bm{\Omega}|}\int_{\Omega}\bm{u}^{\prime}(\bm{x},t)^{T}\bm{u}^{\prime}(\bm{x},t^{\prime})d\bm{\Omega}\,, (50) where \Omega is the domain under investigation and |\Omega| is an appropriate measure of it; this means an area in a 2D domain or a volume in a 3D domain. For a stationary flow, this operator only depends on the time lag between the two instances considered, i.e. K(\tau)=K(t,t+\tau) . In the same way, the eigendecomposition of the covariance matrix in section 3.2 gives information about the leading direction of turbulent mixing, the eigendecomposition of the covariance operators C and K give important information about the existence of coherent patterns in the data. The study of these is the subject of modal analysis in section 6 . Before proceeding with the computation of these operators and their sample definition, it is time to discuss the challenges in treating scattered datasets. 5 Local statistics of a flow field The computation of all quantities described in the previous sections for the case where data is available on a grid is textbook material. We thus start from there in section 5.1 . The computation for the case of scattered data requires additional work and definitions. We first briefly review the traditional approaches in section 5.3 . Then, we take a brief detour into the realm of physics-constrained regression in section 5.3 , before delving into the proposed meshless computation of statistics in section 5.4 . 5.1 Gridded data The data is provided on a structured grid, denoted as \bm{x}_{\bm{i}}=(x_{m},y_{n},z_{l}) , with m=1,\dots,n_{x} , n=1,\dots,n_{y} , and l=1,\dots,n_{z} . Thus, the grid consists of n_{p}=n_{x}n_{y}n_{z} points, which we can index as \bm{i}=1,\dots,n_{p} . For each point, it is possible to define a volume (or area, in 2D) \Delta\Omega_{\bm{i}} based on the half-distance to the neighboring points. It is important to note that the grid does not need to be uniform for all the operations defined in this section. However, we assume that the grid is fixed , meaning that the data is collected on the same points for each snapshot. We assume that these are available on a time discretization \mathbf{t_{k}} , with k=1,\dots n_{t} . Again, this is not necessarily uniformly sampled, but we can define a time interval \Delta t_{k} within which a given snapshot k is “representative”. To ease computations, it is particularly convenient to store all the data in the form of snapshot matrices. For this, we reshape the velocity components at a given snapshot k as a column vector. The associated snapshot matrices are thus \mathbf{U},\mathbf{V},\mathbf{W}\in\mathbb{R}^{n_{p}\times n_{t}} . Each column collects a snapshot at time k and each row contains a time series sampled at the grid point \bm{i} , located at \bm{x}_{i} . All statistics involving the time domain are essentially statistics along the rows of the snapshot matrices. Python offer efficient functions to compute these, as we illustrate in the following exercise. All computations are easy as long as the statistics are computed on the set of grid points in common to all snapshots. The discrete approximation of the integrals for the average and covariances become (see section 3.2 ): \langle\bm{u}\rangle(\bm{x}_{i})=\sum^{n_{t}}_{k=1}\bm{u}(\bm{x}_{i},\mathbf{t_{k}})\frac{\Delta t_{k}}{T}\quad\mbox{and}\quad\langle\bm{u}^{\prime}_{l},\bm{u}^{\prime}_{m}\rangle(\bm{x}_{i})=\sum^{n_{t}}_{k=1}\bm{u}_{l}^{\prime}(\bm{x}_{i},\mathbf{t_{k}})\bm{u}_{m}^{\prime}(\bm{x}_{i},\mathbf{t_{k}})\frac{\Delta t_{k}}{T}\,, (51) with \Delta t_{k}/T=1/n_{t} in case of equally spaced samples in time. In Python, the Reynolds stresses for a 2D flow on uniformly sampled data can be computed as follows ⬇ 1 def compute_row_statistics ( U , V ): 2 # Compute row-wise means 3 mean_U = np . mean ( U , axis =1) 4 mean_V = np . mean ( V , axis =1) 5 # Compute row-wise variances 6 R_uu = np . var ( U , axis =1, ddof =1) 7 R_vv = np . var ( V , axis =1, ddof =1) 8 # Compute row-wise covariances 9 R_uv = np . mean (( U - mean_U [:, None ]) * \ 10 ( V - mean_V [:, None ]), axis =1) 11 return mean_U , mean_V , R_uu , R_vv , R_uv Further processing of the Reynolds stress matrix as described in section 4.1 can be carried out using standard routines as showcased in the following exercise. Finally, the covariance functions in space and time in ( 47 ) and ( 50 ) are now samples on n_{p} and n_{t} points, respectively, and thus become covariance matrices , build out of approximation of the integrals based on the available data. The (scalar) covariance matrix in space, i.e. the discrete version of ( 47 ), is: \mathbf{C}_{s}[\bm{x}_{l},\bm{x}_{m}]=\mathbf{C}_{l,m}=\sum^{n_{t}}_{k=1}\bm{u}^{T}(\bm{x_{l}},\mathbf{t_{k}})\bm{u}(\bm{x_{m}},\mathbf{t_{k}})\frac{\Delta t_{k}}{T}\,. (52) Defining as \mathbf{w}_{k,T}=\Delta t_{k}/T a set of weights, collected on a diagonal matrix \mathbf{W}_{w,T}=\mbox{diag}(\mathbf{w}_{k,T}) , this matrix can be conveniently computed using matrix multiplication from the snapshot matrices as follows: \mathbf{C}_{s}=\mathbf{U}\mathbf{W}_{w}\mathbf{U}^{T}+\mathbf{V}\mathbf{W}_{w}\mathbf{V}^{T}+\mathbf{W}\mathbf{W}_{w}\mathbf{W}^{T}\in\mathbb{R}^{n_{p}\times n_{p}}\,. (53) On the other hand, the discrete version of ( 48 ) reads \mathbf{C}=\begin{bmatrix}\mathbf{U}\mathbf{W}_{w,T}\mathbf{U}^{T}&\mathbf{U}\mathbf{W}_{w,T}\mathbf{V}^{T}&\mathbf{U}\mathbf{W}_{w,T}\mathbf{W}^{T}\\ \mathbf{V}\mathbf{W}_{w,T}\mathbf{U}^{T}&\mathbf{V}\mathbf{W}_{w,T}\mathbf{V}^{T}&\mathbf{V}\mathbf{W}_{w,T}\mathbf{W}^{T}\\ \mathbf{W}\mathbf{W}_{w,T}\mathbf{U}^{T}&\mathbf{W}\mathbf{W}_{w,T}\mathbf{V}^{T}&\mathbf{W}\mathbf{W}_{w,T}\mathbf{W}^{T}\end{bmatrix}\in\mathbb{R}^{3n_{p}\times 3n_{s}}\,. (54) Finally, the temporal correlation matrix ( 50 ) becomes \mathbf{K}=\mathbf{U}^{T}\mathbf{W}_{w,S}\mathbf{U}+\mathbf{V}^{T}\mathbf{W}_{w,S}\mathbf{V}+\mathbf{W}^{T}\mathbf{W}_{w,S}\mathbf{W}\in\mathbb{R}^{n_{t}\times n_{t}}\,, (55) with \mathbf{W}_{w,S}=\mbox{diag}(w_{k,S}) and \mathbf{w}_{k,S}=|\Delta\Omega|_{k}/|\Omega| the weights in space. Exercise 3: Statistics of a Turbulent Flow from Gridded data Consider the PIV data from test case 2.2 . We are interested in computing the (1) the mean flow, (2) the turbulent kinetic energy, (3) the Reynolds stresses and the norm of the anisotropic tensor (4) locate the turbulent states observed in the data in the Lumley triangle. What kind of turbulence is found there? We stress that Reynolds stresses play a crucial role in describing the mixing and momentum transfer that occur within the jet as it spreads into the surrounding fluid. Note that these computations require some important assumption on the flow, since only two out of the three velocity components are measured. We here assume that the flow is perfectly axisymmetric. Therefore, denoting as (u,v,w) the velocity components in cylindrical coordinate along the (x,r,\theta) directions (stream-wise, radial and angular), the Reynolds stress takes the form (see Pope, (2000) ) \bm{R}(\bm{x})=\begin{pmatrix}\langle u^{\prime 2}\rangle(\bm{x})&\langle u^{\prime}v^{\prime}\rangle(\bm{x})&0\\ \langle v^{\prime}u^{\prime}\rangle(\bm{x})&\langle v^{\prime 2}(\bm{x})\rangle(\bm{x})&0\\ 0&0&\langle w^{\prime 2}\rangle(\bm{x})\end{pmatrix}\,, (56) The circumferential symmetry ensures that all cross terms involving azimuthal components are zero. At the centerline, the radial and circumferential components become indistinguishable, allowing the assumption that \langle v\rangle=\langle w\rangle . While this equivalence does not generally hold at greater distances from the centerline, we assume it to be valid for the purposes of this exercise. Solution . Once the data is organized into snapshot matrix, the computation of the mean is trivial. However, note that this datasets contains a large number of NaNs, hence a first logical check on the validity of each vector is performed. The script becomes: ⬇ 1 # Step 1: Mean velocity field 2 # valid vectors 3 valid = np . logical_and ( np . isfinite ( U ), 4 np . isfinite ( V )) 5 # number of valid vectors 6 valid_sum = valid . sum ( axis =0) 7 # perform the average on the valid vectors only 8 U_mean = np . nansum ( U * valid , axis =0) / valid_sum 9 V_mean = np . nansum ( V * valid , axis =0) / valid_sum Figure 9 shows the u and v component of the average velocity field obtained from the PIV data. Figure 9 : Average velocity field from PIV data, u (left) and v (right) components. The computation of the fluctuation component, Reynolds stress tensor, turbulent kinetic energy, anisotropic stress tensor and its norm are given by the following script: ⬇ 1 # Step 2: Turbulent kinetic energy 2 # compute fluctuation fields 3 U_prime = U - U_mean [ np . newaxis , :] 4 V_prime = V - V_mean [ np . newaxis , :] 5 # compute average fluctuations 6 uu_mean = np . nansum ( U_prime * U_prime * valid , axis =0) / 7 ( valid_sum - 1) 8 vv_mean = np . nansum ( V_prime * V_prime * valid , axis =0) / 9 ( valid_sum - 1) 10 uv_mean = np . nansum ( U_prime * V_prime * valid , axis =0) / 11 ( valid_sum - 1) 12 # compute mean TKE 13 fill_value = np . zeros_like ( uu_mean ) 14 # assemble the Reynolds stress tensor 15 R_ij = np . array ([ 16 [ uu_mean , uv_mean , fill_value ], 17 [ uv_mean , vv_mean , fill_value ], 18 [ fill_value , fill_value , vv_mean ] 19 ]) 20 # compute turbulent kinetic energy 21 k = np . sum ( np . diagonal ( R_ij ), axis =2) / 2 22 # Step 3: Anisotropic tensor 23 # components of the anisotropic tensor 24 A_ij = R_ij / (2* k [ np . newaxis , np . newaxis , :]) - 25 np . diag ( np . full (3, 1/3))[:, :, np . newaxis , np . newaxis ] 26 # norm of the anisotropic tensor 27 A_norm = np . linalg . norm ( A_ij , axis = (0,1)) Note that the python object R_ij collects the Reynolds stress tensor in all entries of the domain (this is a tensor of size 3\times 3\times 72\times 71 ). The left side of figure 10 shows the turbulent kinetic energy k and the turbulence intensity to give an idea of the turbulence level. The right part shows the norm of the anisotropic tensor ||\bm{A}|| for PIV data. Figure 10 : Average turbulent kinetic energy k (left), turbulence intensity (center) and norm of the anisotropic tensor (right) from the PIV data. The shear layers in the flow are clearly visible. These are regions of large turbulence intensity and large anisotropicity. Traditional low fidelity turbulence models, based on the Boussinesq Hypothesis and the notion of eddy viscosity, generally have difficulties in describing anisotropicity. These model seek to capture the effects of anisotropicity in an indirect way, namely through the link to the gradients of the mean flow; but such a link does not always hold in complex scenarios ( Pope,, 2000 ; Davidson,, 2004 ) . Finally, the computation required to position the dataset in the invariant map is provided below: ⬇ 1 # Step 4: Lumley triangle 2 # Boundaries of the invariant map 3 x_1C = np . array ([2/3, -1/3, -1/3]) 4 x_2C = np . array ([1/6, 1/6, -1/3]) 5 x_3C = np . array ([0, 0, 0]) 6 7 # II coordinate of the triangle 8 II_1C = x_1C [0]**2 + x_1C [0]* x_1C [1] + x_1C [1]**2 9 II_2C = x_2C [0]**2 + x_2C [0]* x_2C [1] + x_2C [1]**2 10 II_3C = x_3C [0]**2 + x_3C [0]* x_3C [1] + x_3C [1]**2 11 12 # III coordinates of the triangle 13 III_1C = - x_1C [0]* x_1C [1] * ( x_1C [0] + x_1C [1]) 14 III_2C = - x_2C [0]* x_2C [1] * ( x_2C [0] + x_2C [1]) 15 III_3C = - x_3C [0]* x_3C [1] * ( x_3C [0] + x_3C [1]) 16 17 # Number of points to draw the limiting curves 18 n_p = 101 19 20 # Curve from 1 to 3 21 x_13 = np . array ([ 22 np . linspace (0, 2/3, n_p ), 23 np . linspace (0, -1/3, n_p ), 24 np . linspace (0, -1/3, n_p ), 25 ]) 26 27 # Convert into II and III coordinates 28 II_13 = x_13 [0, :]**2 + x_13 [0, :]* x_13 [1, :] + x_13 [1, :]**2 29 III_13 = - x_13 [0, :]* x_13 [1, :] * ( x_13 [0, :] + x_13 [1, :]) 30 31 x_23 = np . array ([ 32 np . linspace (0, -1/3, n_p ), 33 np . linspace (0, 1/6, n_p ), 34 np . linspace (0, 1/6, n_p ) 35 ]) 36 37 II_23 = x_23 [0, :]**2 + x_23 [0, :]* x_23 [1, :] + x_23 [1, :]**2 38 III_23 = - x_23 [0, :]* x_23 [1, :] * ( x_23 [0, :] + x_23 [1, :]) 39 40 x_12 = np . array ([ 41 np . linspace (2/3, 1/6, n_p ), 42 np . linspace (-1/3, -1/3, n_p ), 43 1/3 - np . linspace (2/3, 1/6, n_p ) 44 ]) 45 46 II_12 = x_12 [0, :]**2 + x_12 [0, :]* x_12 [1, :] + x_12 [1, :]**2 47 III_12 = - x_12 [0, :]* x_12 [1, :] * ( x_12 [0, :] + x_12 [1, :]) 48 49 # No need for the 2 last dimensions of the array (x and y dimensions) 50 # A is flattened into a 3x3xn_vectors array 51 a_ij = np . reshape ( A_ij , [3, 3, np . shape ( A_ij )[2]* np . shape ( A_ij )[3]]). T 52 # Compute eigenvalues of the anisotropic tensor 53 eig_vals = np . linalg . eigvals ( a_ij ). T 54 eig_vals = eig_vals - eig_vals . mean ( axis =0)[ np . newaxis , :] 55 eig_vals = np . sort ( eig_vals , axis =0)[::-1, :] 56 57 # Convert to II and III coordinates 58 II = eig_vals [0, :]**2 + 59 eig_vals [0, :] * eig_vals [1, :] + 60 eig_vals [1, :]**2 61 III = - eig_vals [0, :] * eig_vals [1, :] * 62 ( eig_vals [0, :] + eig_vals [1, :]) The resulting Lumley triangle is shown in Figure 11 . Most of the points in the far field and in the jet centerline are close to the \mathbf{x}_{3C} state, hence close to anisotropic turbulene. The region in the shear layer is close to the \bm{x}_{3C}-\bm{x}_{1C} corresponding to an axisymmetric contraction. This state of turbulence occurs when turbulence is “compressed” axially while spreading radially. As a result, the Reynolds stress tensor has nearly equal components on radial and circumferential direction and a smaller component along the axial one. This also occurs in nozzle flows. Figure 11 : Lulmey triangle for the PIV data. 5.2 Traditional binning methods The main challenge with randomly scattered data is that the velocity fields are available at different locations for each snapshot. Let \mathbf{U}^{(j)}(\mathbf{X}^{(j)}) denote the velocity field in snapshot j , with \mathbf{X}^{(j)}=[\mathbf{x}^{(j)},\mathbf{y}^{(j)},\mathbf{z}^{(j)}]\in\mathbb{R}^{n^{(j)}_{p}\times 3} the matrix that collects the coordinate at which the velocity information is available and n^{(j)}_{p} the number of measurement points at each snapshot. It is not rare to have that \mathbf{X}^{(j)}\cap\mathbf{X}^{(l)}=\emptyset for all j,l in n_{t} : in words, not two snapshots share the same sampling location within a dataset of n_{t} snapshot. Considering for example the mean flow and recalling that this is defined as \langle\bm{u}\rangle(\bm{x})=\int^{\infty}_{-\infty}\bm{u}(\bm{x})f_{u}(\bm{x},\bm{u})d\bm{u}\,, (57) with f_{u} the joint probability function assigning a probability to a velocity vector occurring at a given location, not having sufficient samples at every location renders the problem of pdf estimation impractical. The simplest way to compute statistics is to define a grid of "bins", that is areas where statistics are associated with a specific point called the bin center. This underpins the concept of ensemble PTV (EPTV, Kähler et al., 2012 ). Defining as \bm{x_{i}} the location of a bin, one can build the following estimate for the mean flow \langle\mathbf{u}\rangle(\mathbf{x}_{i})\approx\frac{1}{n_{p,i}}\sum^{n_{p,i}}_{i=1}\bm{U}^{(j)}(\mathbf{x}_{i})\,, (58) where \bm{U}^{(j)}(\mathbf{x}_{i}) denotes the mapping of the PTV sample \bm{U}^{(j)} onto the i -th bin, and n_{p,i} denotes the number of measurement points available within the bin. The definition of expectation in this binned formalism can be easily extended to higher order statistics. When a sufficiently large number of particles pass through a bin, the distribution within the bin can approximate the local flow distribution ( Kähler et al.,, 2012 ) . Binning methods differ in how they compute statistics: the "top-hat" approach assigns equal weight to all samples within a bin, while Gaussian weighting ( Agüí and Jiménez,, 1987 ) prioritizes samples closer to the bin center. A significant challenge lies in the propagation of errors or unresolved gradients in the mean flow to higher-order statistics, as highlighted by Agüera et al., (2016) . This issue can be mitigated by employing local polynomial fits for the mean flow. In these notes, we present an alternative approach recently proposed by Ratz and Mendez, (2024) . This method is "meshless," as it avoids the need for a grid to compute derivatives, and "binless," as it eliminates the need for bins to compute local statistics. The idea is to combine the physics-constrained RBF formalism proposed in Sperotto et al., (2022) with an ensemble trick for the regression of flow statistics. Before presenting the method in 5.4 , we propose a brief review of physics-constrained RBF in 5.3 . 5.3 Fundamentals of (constrained) Radial Basis Functions (RBFs) Regression via Radial Basis Functions (RBFs) consists in building a target function from a set of scattered data points using a linear combination of real-valued functions that depends only on the distance from a center point. A classic RBF is the Gaussian 7 7 7 RBFs are usually denoted with \phi or \psi , but in these notes, both symbols are taken for the spatial and temporal structures in modal analysis in the next section. May the readers with experience on RBF forgive us for the use of \gamma ! \gamma(\bm{x}|\bm{x}_{n}^{*},c_{n})=\exp\left(-c_{n}^{2}\left\|\bm{x}-\bm{x}_{n}^{*}\right\|^{2}\right) (59) where c_{n} is the shape parameter and \bm{x}^{*}_{k}\in\mathbb{R}^{d} is the collocation point . The reader is referred to Fornberg and Flyer, (2015) ; Hoffman and Frankel, (2018) ; Buhmann, (2003) ; Trefethen, (2013) for other bases and to learn more about RBFs. The RBF approximation of a generic scalar function f(\bm{x}) , f:\mathbb{R}^{d}\rightarrow\mathbb{R} is therefore: f(\bm{x})=\sum^{n_{b}}_{n=1}\bm{w}_{n}\gamma(\bm{x}|\bm{x}_{n}^{*},c_{n})=\sum^{n_{b}}_{n=1}\bm{w}_{n}\gamma_{n}(\bm{x})\,, (60) having introduced the compact notation for the n-th basis. In traditional RBF regression, both c_{n} and \bm{x}_{n}^{*} are pre-assigned based on the data distribution. This is typically done to ensure that the basis functions are well distributed across the domain and that each basis has sufficient data points within its region of influence 8 8 8 See Sperotto et al., (2022) for a cluster-based approach to the collocation problem. , commonly defined as the area where \gamma_{n}>0.5 . Once all the bases are assigned, the RBF regression consists in identifying the weights of the linear combination \{\bm{w}_{n}\}^{n_{b}}_{n=1} , which we here collect in a vector \mathbf{w}\in\mathbb{R}^{n_{b}} . The key advantage over more sophisticated regression methods (see Mendez, 2024b for a general introduction) is that the function approximation ( 60 ) depends linearly on the parameters \mathbf{w}\in\mathbb{R}^{n_{b}} . Given a set of training data \{\mathbf{x}^{*}_{i},\mathbf{f}_{i}\}^{n_{*}}_{i=1} , which we store in a matrix \mathbf{X}_{*}\in\mathbb{R}^{n_{*}\times d} and a vector \mathbf{f}\in\mathbb{R}^{n_{*}} , these weights are usually computed as those that minimize the following cost function J(\mathbf{w})=||\mathbf{f}-\bm{\Gamma}(\mathbf{X}_{*})\mathbf{w}||^{2}_{2}+\alpha||\mathbf{w}||^{2}_{2}\,, (61) where \bm{\Gamma}(\mathbf{X}_{*})\in\mathbb{R}^{n_{*}\times n_{b}} is the matrix collecting in each column the values of a given basis on the training data \mathbf{X}_{*} and \alpha\in\mathbb{R}^{+} is a user defined parameter controlling the weight of the penalty. In the classic probabilistic interpretation of the regression, the weights minimizing ( 61 ) provide the ‘‘Maximum A Posteriori" (MAP) estimate, that is the most likely set of vectors combining a prior assumption 9 9 9 Here \mathcal{N}(\mathbf{\mu},\bm{\Sigma}) is a multivariate Gaussian with mean \mathbf{\mu} and covariance matrix \bm{\Sigma} , while \bm{I} is the identity matrix of appropriate size. \mathbf{w}\sim\mathcal{N}(\mathbf{0},\alpha^{-1}\bm{I}) with the available data ( Deisenroth et al.,, 2020 ) . This is also known as Ridge Regression. The minimization of ( 61 ) leads to a linear system which offers an analytic solution: \bigl(\bm{\Gamma}^{T}(\mathbf{X}_{*})\bm{\Gamma}(\mathbf{X}_{*})+\alpha\bm{I}\bigr)\mathbf{w}=\bm{\Gamma}^{T}(\mathbf{X}_{*})\mathbf{f}\quad\rightarrow\mathbf{w}=\bigl(\bm{\Gamma}^{T}(\mathbf{X}_{*})\bm{\Gamma}(\mathbf{X}_{*})+\alpha\bm{I}\bigr)^{-1}\bm{\Gamma}^{T}(\mathbf{X}_{*})\mathbf{f}\,. (62) Nevertheless, it is worth pointing out that the inversion in ( 62 ) is numerically inefficient, and in practice the solution of the linear system is better carried out using Cholesky decomposition or the Conjugate Gradient (CG) method ( Trefethen and Bau,, 1997 ) . The linearity with respect to the model parameters (and the resulting quadratic dependency in ( 61 )) facilitates the integration of quadratic penalties and linear constraints into the regression. Denoting these as ||\mathbf{A}_{p}\mathbf{w}||^{2}_{2}\,\quad\mbox{and}\quad\mathbf{A}_{c}\mathbf{w}=\mathbf{c}\,, (63) respectively, with \mathbf{A}_{p}\in\mathbb{R}^{n_{b}\times n_{b}} and \mathbf{A}_{c}\in\mathbb{R}^{n_{\lambda}\times n_{b}} , the penalized and constrained problem is the one that minimizes the augmented cost function: A(\mathbf{w})=J(\mathbf{w})+\alpha_{2}||\mathbf{A}_{p}\mathbf{w}||^{2}_{2}+\bm{\lambda}^{T}\bigl(\mathbf{A}_{c}\mathbf{w}-\mathbf{c})\bigr)\,, (64) where \bm{\lambda}\in\mathbb{R}^{n_{\lambda}} is the vector of (unknown) Lagrange multipliers required to enforce the constraints and \alpha_{2}\in\mathbb{R}^{+} is an additional (user-defined) penalty parameter. The constrained RBF regression now takes the form of a traditional quadratic programming problem ( Nocedal and Wright,, 2006 ; Chong and Zak,, 2013 ) : the solution gives the weights and multipliers [\mathbf{w},\bm{\lambda}] minimizing ( 64 ); by setting the gradient of ( 64 ) with respect to these equal to zero, the minimization leads to a linear system \left(\begin{array}[]{cc}\bm{A}&\bm{B}\\ \bm{B^{T}}&\bm{0}\end{array}\right)\left(\begin{array}[]{c}\bm{w}\\ \bm{\lambda}\end{array}\right)=\left(\begin{array}[]{c}\bm{b}_{1}\\ \bm{b}_{2}\end{array}\right)\,, (65) with \bm{A}=\bm{\Gamma}^{T}\bm{\Gamma}+\alpha_{1}\bm{I}+\alpha_{2}\mathbf{A}_{p}^{T}\mathbf{A}_{p}\in\mathbb{R}^{n_{b}\times n_{b}} , \bm{B}=\mathbf{A}_{c}^{T}\in\mathbb{R}^{n_{b}\times n_{\lambda}} , \bm{b_{1}}=\bm{\Gamma}^{T}\mathbf{f}\in\mathbb{R}^{n_{b}} and \bm{b}_{2}=\mathbf{c}\in\mathbb{R}^{n_{\lambda}} . The reader is referred to Sperotto et al., (2022) and Nocedal and Wright, (2006) for efficient methods to solve ( 65 ). In the context of RBF regression of tracking velocimetry, this framework allows to impose constraints such as boundary conditions (e.g. no slip or symmetry), compliance with sensor data or differential constraints such as divergece-free conditions. A detailed discussion on the constraint implementation is beyond the scopes of this introduction and the reader is referred to Sperotto et al., (2022) for more details. An open-source library implementing this framework, currently under development at the von Karman Institute, has been released in Sperotto et al., 2024b () . We close this section with the RBF regression of a velocity field (i.e. a vector valued function). This is here written as \begin{split}\bm{u}(\mathbf{x},\mathbf{t_{k}})=\begin{pmatrix}u(\mathbf{x})\\ v(\mathbf{x})\\ w(\mathbf{x})\end{pmatrix}=\sum_{n=1}^{n_{b}}\begin{pmatrix}\mathbf{w}_{u,n}(\mathbf{t_{k}})\,\gamma_{n}(\bm{x},\mathbf{t_{k}})\\ \mathbf{w}_{v,n}(\mathbf{t_{k}})\,\gamma_{n}(\bm{x},\mathbf{t_{k}})\\ \mathbf{w}_{w,n}(\mathbf{t_{k}})\,\gamma_{n}(\bm{x},\mathbf{t_{k}})\end{pmatrix}=&\sum_{n=1}^{n_{b}}\begin{pmatrix}\mathbf{w}_{u,n}(\mathbf{t_{k}})\\ \mathbf{w}_{v,n}(\mathbf{t_{k}})\\ \mathbf{w}_{w,n}(\mathbf{t_{k}})\end{pmatrix}\gamma_{n}(\bm{x},\mathbf{t_{k}})\\ &=\sum_{n=1}^{n_{b}}\mathbf{w}_{n}(\mathbf{t_{k}})\gamma_{n}(\bm{x},\mathbf{t_{k}})\,.\end{split} (66) In other words, by taking the same bases for all the components, the RBF regression writes the vector field \bm{u}=(u,v,w)^{T} as a linear combination of vector fields \mathbf{w}_{n}=(\bm{w}_{u,n},\bm{w}_{v,n},\bm{w}_{w,n})^{T} . Equation ( 66 ) still allows the set of bases to change from snapshot to snapshot (hence the time dependence in \gamma_{n} ). As long as no constraints are used that involve the interaction of the components (e.g., divergence-free constraint), the regression of the weights for each component can be carried out independently, ie, setting the problem in ( 62 ) with \mathbf{f} corresponding to the data collected for the three components of the velocity fields. Noticing that only the right-hand side is changing in the three associated systems, it is possible to use a single Cholesky factorization. Finally, note that the RBF regression ( 66 ) produces vector fields which are continuous in space but – at least in the formulation in these notes– are discrete in time. Exercise 4: Unconstrained Regression of a PTV field Consider the synthetic PTV field generated from the test case in Section 2.2 . Perform the regression of a single snapshot using the SPICY package ( Sperotto et al., 2024a, ) . Use semi-random Halton points as collocation and plot the resulting circles and resulting velocity field. Then, using parallel computing, perform a regression of the first 1000 snapshots. Solution . We start by loading the data and creating the SPICY object. Since the regression is unconstrained, we use a ’scalar’ model which reuses the matrix factorizations for U and V . We also save the collocation points and use them in every snapshot. ⬇ 1 # Data loading 2 X_p , Y_p , U_p , V_p = np . genfromtxt ( Name ). T 3 4 # We initialize the Spicy object 5 SP = Spicy ( points =[ X_p , Y_p ], data =[ U_p , V_p ], 6 model = ’scalar’ , verbose =0) 7 # Perform the random collocation. Save these collocation points 8 SP . collocation ( n_K =[5], method = ’semirandom’ , 9 r_mM =[0.001, 50], eps_l =0.8) 10 collocation_path = Fol_Rbf + os . sep + ’RBFs.dat’ # Store the the RBF info 11 np . savetxt ( collocation_path , 12 np . column_stack (( SP . X_C , SP . Y_C , SP . c_k )), 13 delimiter = ’\t’ , fmt = ’%.6g’ ) 14 15 # Assemble and solve the linear system 16 SP . Assembly_Regression () 17 SP . Solve ( K_cond =1 e11 ) 18 19 # Get the solution on a new (arbitrary) grid 20 n_x = 100; n_y = 50 21 x_reg = np . linspace ( np . min ( X_p ), np . max ( X_p ), n_x ) 22 y_reg = np . linspace ( np . min ( Y_p ), np . max ( Y_p ), n_y ) 23 X_reg , Y_reg = np . meshgrid ( x_reg , y_reg ) 24 25 U_reg , V_reg = SP . get_sol ( points =[ X_reg . ravel (), Y_reg . ravel ()], 26 shape = X_reg . shape ) The resulting collocation points are shown in Fig. 12 . Notice that the clustering is very dense with quite large bases. The gradients in the present case are relatively tame, so it is easy to get away with that. Figure 12 : Example of random RBF basis collocation with shape factor controlled by the RBF value on the neighbour basis. This is already it! That is all that is needed to do a regression of data with SPICY. A few functions calls and codes result in the comparison in Fig. 13 which shows side by side the tracking data used for the regression (on the left) and the RBF field on a very fine grid. Figure 13 : PTV snapshot (left) and result from the RBF regression evaluated on a fine grid (right). All the codes producing these images are in Exercise_4.py. The code first performs one regression to generate the individual figures and then uses parallel computing to carry out the regression of 1000 snapshots, which will then be used for the exercise on modal analysis. 5.4 Meshless and binless statistics of scattered data via RBF The concept of regression of instantaneous velocity fields can be extended to the regression of fields of statistical quantities, as proposed by Ratz and Mendez, (2024) . We outline the general idea here and refer the reader to the article for a detailed derivation and example. The main idea is to introduce the RBF regression ( 66 ) into the ensemble expectation operator and leverage the linearity of the regression with respect to the weights. For the mean flow, for example, ( 57 ) becomes: \displaystyle\langle\bm{u}\rangle(\bm{x})=\int^{\infty}_{-\infty}\bm{u}(\bm{x})f_{u}(\bm{x},\bm{u})\text{d}\bm{u}=\bm{\Gamma}(\bm{x})\int^{\infty}_{-\infty}\mathbf{w}f_{w}(\mathbf{w})\text{d}\mathbf{w}=\bm{\Gamma}(\bm{x})\langle\bm{w}\rangle\,, (67) assuming that the position of the bases is maintained over all the snapshots, and having introduced the Jacobian \text{d}\bm{u}/\text{d}\mathbf{w}=\bm{\Gamma}(\mathbf{x}) and the probability density function f_{w}(\mathbf{w})=f_{u}(\mathbf{x},\mathbf{u})\bm{\Gamma}(\bm{x}) . The advantage is that the expectation of the weights can be estimated more easily than the one of the mean velocity field because the distribution f_{w}(\mathbf{w}) does not depend on the spatial position of the data if the velocity fields are sufficiently dense. A first estimate of the weight vector average \langle\mathbf{w}\rangle could be obtained by averaging the weights obtained via regression of all fields: \langle\mathbf{w}\rangle\approx\mathbf{W}_{A}=\frac{1}{n_{t}}\sum_{k=1}^{n_{t}}\mathbf{w}_{k} (68) That is, every snapshot is regressed with the same set of basis and the resulting weights are averaged. However, this requires every snapshot to be sufficiently sampled such that the regression is successful. In practice this is rarely the case because of seeding inhomogeneities. However, introducing ( 62 ) into ( 68 ) leads to interesting avenues for simplifications: \langle\mathbf{w}\rangle\approx\mathbf{w}=\frac{1}{n_{t}}\sum_{k=1}^{n_{t}}\bigl(\bm{\Gamma}^{T}(\mathbf{X}^{(k)})\bm{\Gamma}(\mathbf{X}^{(k)})+\alpha\bm{I}\bigr)^{-1}\bm{\Gamma}^{T}(\mathbf{X}^{(k)})\mathbf{U}^{(k)}\,, (69) where \bm{\Gamma}(\mathbf{X}^{(k)} and \mathbf{U}^{(k)} are the RBF matrix and velocity data in the k-th snapshot. Some of the terms in this summation could be replaced by operations on the ensemble of data points, defined as the union of all the data collected during an acquisition: \mathbf{x}_{E}=\bigcup_{k\in 1\dots n_{t}}\mathbf{X}^{(k)}\qquad\text{and}\qquad\mathbf{U}_{E}=\bigcup_{k\in 1\dots n_{t}}\bm{u}_{k}\,. (70) It is interesting to note that the following relations hold for the covariances and projections: \sum_{k=1}^{n_{t}}\bm{\Gamma}_{k}^{T}\bm{\Gamma}_{k}=\bm{\Gamma}_{E}^{T}\bm{\Gamma}_{E}\qquad\text{and}\qquad\sum_{k=1}^{n_{t}}\bm{\Gamma}_{k}^{T}\bm{U}_{k}=\bm{\Gamma}_{E}^{T}\bm{U}_{E}\,, (71) with \bm{\Gamma}_{E}=\bm{\Gamma}(\mathbf{X}_{E}) . These could be used in ( 69 ) to approximate the average of n_{t} regression with one single regression on the entire ensemble of points. I the snapshots have sufficient seeding, one could further expect the terms \Gamma_{k}^{T}\Gamma to converge towards a common covariance matrix, and thus \bm{\Gamma}_{k}^{T}\bm{\Gamma}_{k}=1/n_{t}\,\bm{\Gamma}_{E}^{T}\bm{\Gamma}_{E} . Inserting this into equation 69 gives another approximation to the ensemble weight: \langle\mathbf{w}\rangle\approx\mathbf{w}_{E}=(\bm{\Gamma}_{E}^{T}\bm{\Gamma}_{E}+\alpha\bm{I})^{-1}\bm{\Gamma}_{E}^{T}\bm{U}_{E}\,. (72) This turns the ensemble of regressions into a regression of the ensemble. While the cost in assembling the linear system is much larger, we only need to solve it once instead of n_{t} times if each snapshot is regressed separately. Moreover, this reduces demands for seeding in the instantaneous fields. Of course, ( 72 ) and ( 69 ) could be combined in an aggregative approach: for example computing the averages from ( 72 ) for a set of n_{E} ensemble and then averaging the results with ( 69 ). The approach also extends effortlessly to higher order statistics since the analytical expression for the velocity field can be used to subtract the computed mean in every point of the ensemble. This field of fluctuations can then be used to compute the (scattered) expression of the field \bm{u}^{\prime}_{i}\bm{u}^{\prime}_{j} which can be regressed to obtain the Reynolds stresses. Again, the reader is referred to Ratz and Mendez, (2024) for a more detailed derivation. Exercise 5: Statistics of a Turbulent Flow from Scattered data Consider the PIV data from test case 2.2 . We are interested in computing the (1) the mean flow, (2) the turbulent kinetic energy, (3) the Reynolds stresses and the norm of the anisotropic tensor (4) locate the turbulent states observed in the data in the Lumley triangle. What kind of turbulence is found there? Use the same assumptions as for the statistics from PIV! Hint: Using refinement regions greatly helps to reduce computational load in both the training data and the clustering. Solution . We start by pruning parts of the training data. While the seeding in the region of interest is relatively uniform, the particles outside of the jet contain more or less the same information since the flow is laminar in these regions. In contrast, the flow in the shear layer has the highest turbulence intensity so every particle counts. We start with 750\,000 from the entire dataset to illustrate the exercise. More particles are possible but this increases memory demands. We use the shapely object to define multiple refinement regions, one set of regions to prune particles and one to refine the collocation in specific areas. ⬇ 1 refinement_1 = np . array ([ 2 [ x_min , x_max , x_max , x_min ], 3 [105, 115, 23, 35], 4 ]) 5 # We use shapely to define the areas 6 polygon_points_1 = geometry . Polygon ( refinement_1 . T ) 7 # Extract the sets of points outside the jet 8 X_out_glo = X [~ in_polygon_1 ] 9 Y_out_glo = Y [~ in_polygon_1 ] 10 U_out_glo = U [~ in_polygon_1 ] 11 V_out_glo = V [~ in_polygon_1 ] Figure 14 shows the three refinement regions. Blue for outside the jet, orange for the shear layer and green for the core of the jet. We then continue by removing a percentage of particles in these areas. This is simply done with a random sample algorithm. Figure 14 : Refinement regions used for particles pruning ⬇ 1 # Here, we do the pruning in each region. These are the fractions 2 # of particles which are kept in each area 3 fraction_out = 0.6 4 fraction_shear = 1.0 5 fraction_core = 0.4 6 7 idcs_out = np . arange ( X_out_glo . shape [0]) 8 np . random . shuffle ( idcs_out ) 9 idcs_out = idcs_out [: int ( X_out_glo . shape [0] * fraction_out )] 10 X_out = X_out_glo [ idcs_out ]; Y_out = Y_out_glo [ idcs_out ] 11 U_out = U_out_glo [ idcs_out ]; V_out = V_out_glo [ idcs_out ] We then define refinement regions in which we want to cluster more finely. These should be similar to our pruning areas. The result is shown in Fig. 15 . Note that the SPICY code plots the basis in these refinement areas if you want to take a look at it. Note that we had to give these refinement areas in normalized coordinates. Depending on your domain, rescaling the longest axis between 0 and 1 can be beneficial. Figure 15 : Refinement regions used for the clustering approach. We continue with the use of the SPICY class ( Sperotto et al., 2024a, ) to compute the regression. You are already experts from exercise 3, so we just highlight the difference of the rescaled domain and refinement. ⬇ 1 # Average number of particles per basis in each refinement level 2 refines = [150, 150, 450, 1500] 3 eps_l = 0.9 4 SP = Spicy ( 5 points =[ 6 ( X_train - x_min ) / scaling , 7 ( Y_train - y_min ) / scaling 8 ], 9 data =[ U_train , V_train ], 10 basis = ’gauss’ , 11 model = ’scalar’ 12 ) 13 SP . collocation ( 14 n_K = refines , 15 Areas =[ poly_refinement_1 , poly_refinement_2 , 16 poly_refinement_3 , None ], 17 r_mM =[0.05, 0.8], 18 eps_l = eps_l 19 ) 20 # Visualize the refined bases 21 SP . plot_RBFs ( level =0) # acts on first shear layer 22 SP . plot_RBFs ( level =1) # acts on second shear layer 23 SP . plot_RBFs ( level =2) # acts on core 24 SP . plot_RBFs ( level =3) # acts everywhere The resulting mean flow field is shown in Fig. 16 . The results are similar to those obtained with the gridded case in the previous exercise. However, the outcome of the RBF is a continuous function, which could be evaluated at any arbitrary grid (thus achieving super-resolution) and could provide analytic derivatives at any arbitrary point. Figure 16 : Resulting mean field for U (left) and V (right) for the meshless regression of the mean flow. We continue by subtracting the (analytical) mean in every point and computing the products of scattered correlations which go into the second regression. ⬇ 1 # Compute the mean in every point and subtract it 2 U_mean_train , V_mean_train = SP . get_sol ( 3 points =[ 4 ( X_train - x_min ) / scaling , 5 ( Y_train - y_min ) / scaling 6 ] 7 ) 8 u_train_prime = U_train - U_mean_train 9 v_train_prime = V_train - V_mean_train 10 # Compute the field of correlations 11 uu_train = u_train_prime * u_train_prime 12 vv_train = v_train_prime * v_train_prime 13 uv_train = u_train_prime * v_train_prime 14 15 # SPICY object for statistics 16 SP_stat = Spicy ( 17 points =[ 18 ( X_train - x_min ) / scaling , 19 ( Y_train - y_min ) / scaling 20 ], 21 data =[ uu_train , vv_train , uv_train ], 22 basis = ’gauss’ , 23 model = ’scalar’ 24 ) 25 # Here, we could also reuse the collocation points since 26 # they are the same 27 SP_stat . collocation ( 28 n_K = refines , 29 Areas =[ poly_refinement_1 , 30 poly_refinement_2 , poly_refinement_3 , None ], 31 r_mM =[0.05, 0.8], 32 eps_l = eps_l 33 ) 34 SP_stat . Assembly_Regression () 35 # We borrow the cholesky factorization from the mean flow since 36 the training data is the same 37 # Careful! SPICY rescales your data range internally, 38 so we need to adapt this 39 SP_stat . L_A = SP . L_A 40 SP_stat . b_1 = SP_stat . b_1 * SP_stat . scale_U / SP . scale_U Figure 17 : Average turbulent kinetic energy k (left), turbulence intensity (center) and norm of the anisotropic tensor (right) from the PTV data. Figure 18 : Lulmey triangle for the PTV data. The rest of the computation is the same as for the PIV since we reuse the same points for plotting. The results in terms of turbulence properties agree with those of the previous exercise. However, the RBF now provides analytical functions for statistics. Moreover, we stress that this regression was carried out without physical constraints, which can significantly accelerate statistical convergence. 6 Global statistics and modal decompositions Data-driven modal analysis is a subset of machine learning and signal processing focusing on decomposing data as a linear combination of simpler components called modes. A broad overview of different methods is provided in Taira et al., (2017) ; Mendez, (2023) , while Mendez, 2024a () connects these to non-linear approaches. The reader is referred to these references, along with the works cited therein, for more details. These notes focus on the implications for computations on gridded versus scattered data. The key difference over more traditional decompositions, such as Fourier or Wavelet decompositions, is that the bases for the modes are tailored to the dataset and not defined a priori. For a velocity field, the general modal decomposition can be written as \bm{u}(\bm{x},t)\approx\bm{u}_{n_{r}}(\bm{x},t)=\sum^{n_{r}-1}_{r=0}\sigma_{r}\bm{\phi}_{r}(\bm{x}){\psi}_{r}(t)\, (73) where \bm{\phi}_{r} and {\psi}_{r} are the spatial and temporal structures of the r -th mode and \sigma_{r} the associated amplitude. Different decompositions are obtained by setting different constraints on either \bm{\phi}_{r} or {\psi}_{r} . In the following, we focus on the Proper Orthogonal Decomposition (POD), the fundamental decompositions derived from all other decompositions. We first start reviewing the POD for continuous data in section 6.1 and its implementation for gridded (section 6.2 ) and scattered (section 6.3 ) data using the RBF-based meshless approach. Finally, section 6.4 closes with some ideas to generalize the meshless approach to other decompositions. 6.1 The (continuous) Proper Orthogonal Decomposition (POD) Let us consider the case in which both the spatial domain \bm{x}\subseteq\Omega and the time domain t\in[0,T] are continuous variables and hence both the "data" \bm{u}(\bm{x},t) and the structures \bm{\phi}_{r}(\bm{x}) and \bm{\psi}_{r}(t) in ( 73 ) are continuous functions, with \bm{\phi}_{r}(\bm{x}) being flow fields. Denoting as {\bm{u}}_{n_{r}}(\bm{x},t) an approximation of the data using n_{r} modes in ( 73 ), the POD is the decompositions defined to minimize the l_{2} error: \mathcal{E}(n_{r})=\frac{1}{T|\bm{\Omega}|}\int_{T}\int_{\bm{\Omega}}\bigl(\bm{u}(\bm{x},t)-{\bm{u}}_{n_{r}}(\bm{x},t)\bigr)^{2}dtd\bm{\Omega}\, (74) The solution to this problem leads to the definition of spatial and temporal structures as eigenfunctions of the covariance functions ( 48 ) and ( 50 ), respectively. These are solutions of the two Fredholm integral eigenvalue problems: \sigma^{2}_{r}\bm{\phi}(\bm{x})=\frac{1}{|\bm{\Omega}|}\int_{\bm{x}^{\prime}\subseteq\Omega}\bm{C}(\bm{x},\bm{x}^{\prime})\bm{\phi}(\bm{x}^{\prime})d\bm{\Omega}\quad\mbox{and}\quad\sigma^{2}_{r}\psi(t)=\frac{1}{T}\int^{T}_{0}K(t,t^{\prime}){\psi}(t^{\prime})dt^{\prime}\,. (75) It is interesting to note that the integrals in ( 74 ) and in ( 75 ) are based on the notion of continuous inner products. These are the inner products in space and in time, defined between functions as \langle a(\bm{x})b(\bm{x})\rangle_{\Omega}=\frac{1}{|\bm{\Omega}|}\int_{\bm{x}^{\prime}\subseteq\Omega}a(\bm{x}^{\prime})b(\bm{x}^{\prime})d\bm{x}^{\prime}\quad\mbox{and}\quad\langle a(t)b(t)\rangle_{T}=\frac{1}{T}\int^{T}_{0}a(t^{\prime})b(t^{\prime})dt^{\prime}\,. (76) Note that these were also used in the definition of functions ( 48 ) and ( 50 ). The main implication in deriving algorithms for computing POD on gridded or scattered data is in how the integrals in the inner products and the definitions of the covariance functions are approximated. 6.2 The POD of gridded data Consider the case of grid data as introduced in section 5.1 . The simplest and most popular approach to approximate the inner product in space in ( 76 ) is the midpoint rule, hence: \langle a(\bm{x})b(\bm{x})\rangle_{\Omega}=\frac{1}{|\bm{\Omega}|}\int_{\bm{x}^{\prime}\subseteq\Omega}a(\bm{x})b(\bm{x})d\bm{x}^{\prime}\approx\sum^{n_{s}-1}_{j=0}a(\bm{x}_{j})b(\bm{x}_{j})\frac{d\Omega_{j}}{|\bm{\Omega}|}=\mathbf{b}^{T}\bm{W}_{s,w}\mathbf{a}\,, (77) having introduced a weight matrix similar to equation ( 55 ) and the sample vectors \mathbf{a},\mathbf{b}\in\mathbb{R}^{n_{t}} collecting samples of the functions a(\bm{x}),b(\bm{x}) on n_{s} points. In this setting, the space eigenvalue problem in ( 76 ) becomes a matrix eigenvalue problem: (\mathbf{C}\mathbf{W}_{s,w})\bm{\phi}_{r}=\sigma^{2}_{r}\bm{\phi}_{r}, (78) where \bm{\phi}_{r}\in\mathbb{R}^{n_{s}} is the sampled r-th eigenfunctions in space. The reader is referred to Sun et al., (2015) ; Kumar et al., (2009) for more quadrature approaches to approximate the integral. Similarly, the approximation for the inner product in time in ( 76 ) becomes \langle a(t)b(t)\rangle_{T}=\frac{1}{T}\int^{T}_{0}a(t)b(t)dt^{\prime}\approx\sum^{n_{t}-1}_{j=0}a(t_{j})b(t_{j})\frac{\Delta t_{j}}{T}=\mathbf{b}^{T}\bm{W}_{t,w}\mathbf{a}\,, (79) and the time eigenvalue problem in ( 76 ) becomes (\mathbf{K}\mathbf{W}_{t,w})\bm{\psi}_{r}=\sigma^{2}_{r}\bm{\psi}_{r}\,, (80) with \psi_{r}\in\mathbb{R}^{n_{t}} is the r-th sampled eigenfunction in time. By definition, the eigenvectors obtained in ( 78 ) and ( 80 ) are orthonormal according to the weighted inner product in ( 77 ) and ( 79 ). The main implication is that it is easy to compute \phi_{r}^{\prime}s from \psi_{r}^{\prime}s and vice versa. The discrete version of ( 73 ) now becomes a matrix factorization: \bm{u}(\bm{x}_{i},\mathbf{t_{k}})=\sum^{n_{r}-1}_{r=0}\sigma_{r}\bm{\phi}_{r}(\bm{x}_{i})\psi^{T}_{r}(\mathbf{t_{k}})\,, (81) having reshaped each snapshot of the velocity field and each spatial structure of the modes into a column vector. The (weighted) time inner product of the expansion ( 81 ) by \psi_{r} reads: \langle\bm{u}(\bm{x}_{i},\mathbf{t_{k}}),\psi_{r}(\mathbf{t_{k}})\rangle_{T}=\Bigg\langle\sum^{n_{r}-1}_{r=0}\sigma_{r}\bm{\phi}_{r}(\bm{x}_{i})\psi^{T}_{r}(\mathbf{t_{k}}),\psi_{r}(\mathbf{t_{k}})\Bigg\rangle_{T}=\sigma_{r}\bm{\phi}_{r}(\bm{x_{i}})\,, (82) so the both the inner decomposition ( 81 ) can be completed. Note that, in the case \bm{W}{s,w} and \bm{W}_{t,w} become identity matrices (i.e., the data is uniformly sampled in space and time), the POD can be computed as the Singular Value Decomposition of the snapshot matrix (see Dawson, (2023) for more details). The most popular POD algorithm, due to Sirovich, (1987) , computes the POD by first assembling and solving the eigenvalue problem in ( 80 ), and then computing the spatial structures via the projection in ( 82 ). 6.3 The meshless POD of scattered data Consider now the case where all the n_{t} datasets have been written as linear combinations of RBFs as ( 66 ). We thus have a finite set of continuous velocity fields in \bm{x}\in\Omega\subset\mathbb{R}^{d} . The inner products in time are discrete, while those in space are continuous. Therefore, following the traditional snapshot POD approach, the eigenvalue in time remains discrete as in ( 80 ). Still, both the definition of the temporal correlation matrix and the projection to identify the spatial structures must change. Building adaptations of these takes us to meshless POD, introduced recently by Tirelli et al., (2024) . A similar idea is known in functional analysis as Functional Principal Component Analysis ( Ramsay and Silverman,, 2005 ; Ramsay and Silverman,, 1997 ; Wang et al.,, 2016 ; Hall and Horowitz,, 2006 , FPCA,) . FPCA generalizes the traditional PCA to functional data, usually taking the form of continuous functions over time. FPCA uses discrete inner products to build a continuous eigenvalue problem, while the proposed approach uses a continuous inner product to build a discrete eigenvalue problem. A brief discussion on the difference between the two is provided in Tirelli et al., (2024) . The idea is to use the RBF expansion within the definition of the temporal correlation matrix. This provides an analytical expression for the integrand in ( 50 ), which can be evaluated at any arbitrary point, enabling location-based quadrature methods, such as the Gauss-Legendre quadrature. However, in these notes, we take an alternative approach: we derive an explicit expression for the temporal correlation matrix by substituting the RBF expansion ( 66 ) into the definition of the temporal covariance function ( 50 ) and formulating the result in terms of the RBF weights. Using the index l\in\{u,v,w\} to span the velocity components and the RBF weight vectors for each velocity component, the result reads: \begin{split}\mathbf{K}_{i,j}&=\frac{1}{|\bm{\Omega}|}\int_{\Omega}\bm{u}^{\prime}(\bm{x},t_{i})^{T}\bm{u}^{\prime}(\bm{x},t_{j})d\bm{\Omega}=\frac{1}{|\bm{\Omega}|}\int_{\Omega}\sum^{3}_{l=1}\bm{u}_{l}^{\prime}(\bm{x},t_{i})\bm{u}_{l}^{\prime}(\bm{x},t_{j})d\bm{\Omega}\\ &=\frac{1}{|\bm{\Omega}|}\int_{\Omega}\sum^{3}_{l=1}\biggl(\sum_{n=1}^{n_{b}}\mathbf{w}_{l,n}(\mathbf{t_{i}})\gamma_{n}(\bm{x},\mathbf{t_{i}})\biggr)\biggl(\sum_{m=1}^{n_{b}}\mathbf{w}_{l,m}(\mathbf{t_{j}})\gamma_{m}(\bm{x},\mathbf{t_{j}})\biggr)d\bm{\Omega}\\ &=\sum^{3}_{l=1}\sum_{n=1}^{n_{b}}\sum_{m=1}^{n_{b}}\mathbf{w}_{l,n}(\mathbf{t_{i}})\mathbf{w}_{l,m}(\mathbf{t_{j}})\biggl(\frac{1}{|\bm{\Omega}|}\int_{\Omega}\gamma_{n}(\bm{x},\mathbf{t_{j}})\gamma_{m}(\bm{x},\mathbf{t_{j}})d\bm{\Omega}\biggr)\\ &=\mathbf{w}^{T}_{u}(\mathbf{t}_{i})\mathbf{I}(\mathbf{t}_{i},\mathbf{t}_{j})\mathbf{w}_{u}(\mathbf{t}_{j})+\mathbf{w}^{T}_{v}(\mathbf{t}_{i})\mathbf{I}(\mathbf{t}_{i},\mathbf{t}_{j})\mathbf{w}_{v}(\mathbf{t}_{j})+\mathbf{w}^{T}_{w}(\mathbf{t}_{i})\mathbf{I}(\mathbf{t}_{i},\mathbf{t}_{j})\mathbf{w}_{w}(\mathbf{t}_{j})\,,\end{split} (83) having introduced the matrix: \mathbf{I}_{m,n}(\mathbf{t}_{i},\mathbf{t}_{j})=\frac{1}{|\bm{\Omega}|}\int_{\Omega}\gamma_{n}(\bm{x},\mathbf{t_{i}})\gamma_{m}(\bm{x},\mathbf{t_{j}})d\bm{\Omega}\,\in\mathbb{R}^{n_{b}\times n_{b}}. (84) If the basis functions differ between snapshots, this matrix will depend on each specific pair of snapshots (i,j) . However, if a common basis is used across all snapshots, only a single matrix, which can be precomputed, is needed. This significantly reduces the computational cost of evaluating ( 83 ). The proposed approach is numerically less accurate than the quadrature-based approach described in Tirelli et al., (2024) . However, it is computationally cheaper and better adapts to complex geometries. Finally, the calculation of the spatial structures following ( 82 ) leads to the expansion of the RBF of the spatial structures \phi_{r}(\bm{x}) : \sigma_{r}\phi_{r}(\bm{x})=\langle\bm{u}(\bm{x},\mathbf{t}_{k}),\psi_{r}(\mathbf{t}_{k})\rangle_{T}=\sum^{n_{b}}_{n=1}\langle\mathbf{w}_{n}(\mathbf{t}_{k}),\psi_{r}(\mathbf{t}_{k})\rangle_{T}\,\gamma_{n}(\bm{x})\,. (85) Exercise 6: Meshless POD vs Gridded POD Consider the PIV data from test case 2.2 , resampled to produce scattered data as in PTV. As for exercise 2, consider only the portion of the data with \mathbf{t}_{k} , that is, during the first stationary condition. The PIV dataset is sufficiently dense to provide a good decomposition (see Mendez et al., (2020) ); The scope of this exercise is to test the meshless POD and see if the obtained modes agree with the gridded ones. For the comparison, we first perform the traditional (grid based) POD of the PIV dataset following Section 6.2 . We compute modal amplitudes, spatial structures and the frequency spectra of the temporal structures for the first three modes. Then, the computation for the meshless approach will be repeated, and the results will be compared. The reader is encouraged to decrease the seeding density in the data generation to see when the decomposition fails. Solution . The POD of gridded data is relatively straightforward since it only involves matrices and matrix multiplications. To perform it, we use the open-source code MODULO ( Poletti et al.,, 2024 ) . The code requires the data matrix \bm{D} of all the stacked snapshots as input. This can be loaded using parallel computing and then passed to MODULO as follows: ⬇ 1 # Function to load the data 2 def load_piv_data ( file_name ): 3 data = np . genfromtxt ( file_name )[1:, :] 4 return data 5 6 # Parallel processing to load the files 7 num_workers = 4 8 with ThreadPoolExecutor ( max_workers = num_workers ) as executor : 9 results = np . array ( list ( executor . map ( lambda file_name :\ 10 load_piv_data ( Fol_Piv + os . sep + file_name ), file_names ))) 11 12 # Reshape into size (n_s, n_t) 13 D = np . transpose ( results , axes =(0, 2, 1)). reshape ( n_t , n_s ). T 14 15 # Import the Modulo package and perform decomposition 16 from modulo_vki import ModuloVKI 17 modu = ModuloVKI ( data = np . nan_to_num ( D ), n_Modes =1000) 18 Phi_grid , Psi_grid , Sigma_grid = modu . compute_POD_K () Note that MODULO computes the POD decomposition in one line (line 18). We compute 50 modes even if only 3 are asked to observe the decomposition convergence. Figure 19a) shows the convergence of the modes, that is the normalized amplitude as a function of the mode index. Figure 19b) shows the frequency content on the leading modes. (a) (b) (c) Figure 19 : Results from the traditional POD of PIV data. Fig (a): Energy \sigma of the first 50 POD modes; Fig (b): spectra of the temporal modes \psi Fig (c): contour of the spatial structure of the first 3 POD modes (U component on the left, V component on the rigth). Figure 19c) shows the contour plots of the velocity fields associated with the first three modes, i.e. \bm{\phi}_{1}(\bm{x}) , \bm{\phi}_{2}(\bm{x}) , \bm{\phi}_{3}(\bm{x}) . Since the data is in stationary conditions and has a solid statistical convergence, the first mode corresponds to the mean flow (note that this is generally not the case for the POD; see Mendez et al., (2019) ). The second and third modes are paired to describe the traveling wave pattern in the shedding (see Mendez et al., (2020) ). These modes have the same amplitude but are in quadrature in space and time (see Barreiro-Villaverde et al., (2021) for methods to identify traveling patterns) The leading frequency in both modes corresponds to the vortex shedding frequency. Combining the leading wavelength on the spatial structures with the leading frequencies in the temporal one, it is possible to estimate the advection speed of these vortices. Moving onto the meshless POD, we first need to build the matrix \mathbf{I}_{m,n} ( 84 ), which requires an integral. Having used the same RBF basis for all snapshots, this matrix must be evaluated only once. For simplicity, we sample it on a domain and use a summation to save the computational cost of a more precise integration rule; note, however, that this integral could be carried out with the Montecarlo method in case of more complex geometries and is the only step that requires considerations on the shape of the domain. The definition of integrand, as a function of the RBF parameters, is provided by the function func while the integration is carried out using a nested for loop. This could be significantly accelerated using parallel computing; however, given the small cost of the operation, we keep it as a simple nested loop. ⬇ 1 2 #%% Meshless POD, computation of I matrix 3 # Input folder of the RBF weights 4 Fol_Rbf = ’RBF_DATA_CYLINDER’ 5 weight_list = sorted ([ file for file in os . listdir ( Fol_Rbf ) 6 if ’RBF’ not in file ]) 7 8 # Function for the integrand in equation 77 9 def func ( x , y , x_c_n , x_c_m , y_c_n , y_c_m , c_n , c_m ): 10 return np . exp (- c_n **2 * (( x - x_c_n )**2 + ( y - y_c_n )**2)) * \ 11 np . exp (- c_m **2 * (( x - x_c_m )**2 + ( y - y_c_m )**2)) 12 13 # load the RBF data 14 X_C , Y_C , c_k = np . genfromtxt ( Fol_Rbf + os . sep + ’RBFs.dat’ ). T 15 n_b = c_k . shape [0] 16 17 # Integration domain (for classic integration) 18 x_integrate = np . linspace ( Xg . min (), Xg . max (), 151) 19 y_integrate = np . linspace ( Yg . min (), Yg . max (), 61) 20 X_integrate , Y_integrate = np . meshgrid ( x_integrate , y_integrate ) 21 X_integrate = X_integrate . ravel () 22 Y_integrate = Y_integrate . ravel () 23 24 # We compute a single matrix I since the RBFs do not change 25 in between time steps , only their weights . 26 # This allows to save time because we use a single I matrix 27 instead of 1000 28 29 I_meshless = np . zeros (( n_b , n_b )) 30 for m in tqdm ( range ( n_b ), mininterval =1, desc = ’Filling I matrix’ ): 31 for n in range (0, n_b ): 32 I_meshless [ m , n ] = func ( X_integrate , Y_integrate , 33 X_C [ n ], X_C [ m ], Y_C [ n ], Y_C [ m ], c_k [ n ], c_k [ m ]). sum () 34 # Divide by the area to normalize the integral 35 area = ( Xg . max () - Xg . min ()) / ( Yg . max () - Yg . min ()) 36 I_meshless = I_meshless / area Since \mathbf{I}_{m,n} is a constant in between all snapshots, we just need to multiply it with the individual weights at \mathbf{t}_{i} and \mathbf{t}_{j} to get the correlation matrix \mathbf{K} ( 83 ). The necessary weights of the first 1000 snapshots were already computed in exercise 3. We here use parallel computing to prepare the matrix K while taking advantage of its symmetry: ⬇ 1 # Function to compute a single element of the covariance matrix 2 def compute_K_element ( i , j , w_U_all , w_V_all , I_meshless ): 3 w_U_i = w_U_all [ i ]; w_V_i = w_V_all [ i ] 4 w_U_j = w_U_all [ j ]; w_V_j = w_V_all [ j ] 5 # Calculate the element of the covariance matrix K[i, j] 6 K_value = w_U_i . T @ I_meshless @ w_U_j + 7 w_V_i . T @ I_meshless @ w_V_j 8 return ( i , j , K_value ) 9 10 # Load all weights ahead of time to avoid repeated I/O 11 w_U_all = []; w_V_all = [] 12 13 # Data Reader 14 for i in tqdm ( range ( len ( weight_list )), desc = ’Loading weights’ ): 15 w_U_i , w_V_i = np . genfromtxt ( Fol_Rbf + os . sep 16 + weight_list [ i ]). T 17 w_U_all . append ( w_U_i ) 18 w_V_all . append ( w_V_i ) 19 20 # Stack weights for easy indexing 21 w_U_all = np . stack ( w_U_all ) 22 w_V_all = np . stack ( w_V_all ) 23 24 # Number of snapshots 25 n_t = len ( weight_list ) 26 27 # Use joblib to parallelize the double loop calculation 28 results = Parallel ( n_jobs =-1)( 29 delayed ( compute_K_element )( i , j , w_U_all , w_V_all , I_meshless ) 30 for i in tqdm ( range ( n_t ), desc = "Computing covariance matrix" ) 31 for j in range ( i + 1) 32 #(Only compute for j <= i since K is symmetric) 33 ) 34 35 # Create an empty covariance matrix 36 K_meshless = np . zeros (( n_t , n_t )) 37 38 # Fill in the covariance matrix with the results from 39 the parallel computation 40 for i , j , value in results : 41 K_meshless [ i , j ] = value 42 K_meshless [ j , i ] = value # Since K is symmetric To minimize the I/O overhead, all weights are first loaded and appended into two lists. The function “compute_K_element” implements ( 83 ), and lines 28-32 distribute the job to various processors. The last step builds the matrix \mathbf{K} . The eigendecomposition of \mathbf{K} can now be computed as in the gridded case to produce the amplitudes and temporal bases (see the provided codes). Finally, to compute the spatial basis, we compute the projections from ( 85 ):S ⬇ 1 2 # The Gamma matrix is the same at every step 3 Gamma = Phi_RBF_2D ( Xg . ravel (), Yg . ravel (), X_C , Y_C ,\ 4 c_k , basis = ’gauss’ ) 5 for i in range ( n_modes ): 6 weights_U_projected = np . squeeze ( 7 Psi_meshless [:, i ][ np . newaxis , :]. dot ( w_U )) 8 weights_V_projected = np . squeeze ( 9 Psi_meshless [:, i ][ np . newaxis , :]. dot ( w_V )) 10 11 Phi_meshless [: nxny , i ] = Gamma . dot ( weights_U_projected )\ 12 / Sigma_meshless [ i ] 13 Phi_meshless [ nxny :, i ] = Gamma . dot ( weights_V_projected )\ 14 / Sigma_meshless [ i ] We conclude by plotting the same quantities as in the gridded case in Figure 20. The agreement with the grid-based POD is remarkable, although the RBF configuration used in this exercise encounters some challenges in regions with sizeable mean flow gradients. A more refined RBF collocation strategy could improve accuracy in these areas. Nevertheless, for this exercise, the comparison is enough to demonstrate the validity of the approach. It is worth emphasizing that this method can be easily applied to more complex geometries, and the resulting modes are analytic in space, offering super-resolution and enabling the analytic computation of derivatives. (a) (b) (c) Figure 20 : Meshless POD. Fig (a): Energy \sigma of the first 50 POD modes; Fig (b): spectra of the temporal modes \psi 6.4 Generalizations beyond the POD All linear decompositions are based on inner products in space and time ( Mendez,, 2023 ) . These solely differ in how the temporal or the spatial structures are computed. Therefore, using the inner products introduced in the previous section in the traditional decompositions produces the meshless formulation for scattered data. Below, we briefly comment on how this could be done for the most common Spectral POD by Sieber et al., (2016) , the Spectral POD by Towne et al., (2018) , the Multiscale POD by Mendez et al., (2019) and the Dynamic Mode Decomposition by Schmid, (2010) . The reader is referred to Mendez, 2024a () for a comprehensive introduction to all these decompositions, all available in the MODULO software package MODULO ( Poletti et al.,, 2024 ) . The meshless implementation of all these decompositions is a work in progress! 6.4.1 The meshless version of Sieber et al., (2016) ’s Spectral POD Sieber et al., (2016) ’s Spectral POD is a variant of the traditional POD that aims to improve the spectral purity of the POD for the case of stationary data. The idea is to apply a diagonal filter to the temporal covariance matrix \mathbf{K} that forces the temporal structure of the POD to have a narrower spectrum. This allows for circumventing pathological conditions in datasets where the energy optimality is more a limitation than an advantage, for example, if features characterized by vastly different scales (i.e., frequency content) have comparable energy contributions to the data. In these cases, the POD modes tend to mix different features in the same modes, which limits the identification of the features. The derivation of the meshless SPOD is straightforward because this decomposition adds only one additional step to the traditional POD: the filtering operation. This is carried out on the matrix \mathbf{K} , which would simply be replaced by the same meshless version used in the meshless POD. Similarly, the projection for the spatial structures remains identical to that for the meshless POD. 6.4.2 The meshless version of Towne et al., (2018) ’s Spectral POD Towne et al., (2018) ’s Spectral POD generalizes Welch’s method to modal decompositions. The idea is to first compute the modal discrete Fourier Transform (DFT) over different chunks of the data in time. Then, the average frequency content in each chunk is replaced by a POD carried out on a snapshot matrix obtained by looking at the evolution of the DFT over the chunks. The result is a large set of harmonic modes. To obtain the meshless version of this decomposition, one must combine the meshless version of the DFT with the meshless POD. The meshless DFT is obtained by combining the RBF expansion in space with the Fourier Transform of the RBF’s weights. The meshless POD can then be applied for each leading frequency to obtain the SPOD modes. 6.4.3 The meshless version of Mendez et al., (2019) ’s Multiscale POD Mendez et al., (2019) ’s Multiscale POD combines the concept of Multiresolution Analysis (MRA) with POD. The mPOD basis is optimal for a given partition of the frequency domain, effectively constraining the frequency content of each mode to a specific range (scale). This frequency decomposition uses Wavelets or filter banks, ensuring the modes remain orthogonal in the time domain. Since MRA is applied to the temporal correlation matrix \mathbf{K} , the derivation of meshless mPOD follows the same procedure as the meshless POD, with the MRA steps remaining unchanged. 6.4.4 The meshless version of Schmid, (2010) ’s DMD Schmid, (2010) ’s Dynamic Mode Decomposition consists of decomposing the data as a linear combination of complex exponentials. This implies approximating the data with a linear dynamical system and fitting its eigenvalues using a least-square approach. Since fitting the linear dynamical system directly to the dataset (that is, to all the time series at each location) is computationally prohibitive, the idea is to fit it on a subset of leading POD modes. Given the complex harmonics that best describe the evolution of the POD modes, it is possible to compute the associated spatial structures via projection in space. The meshless DMD, therefore, can be built from the meshless POD with least-square regression of complex exponentials on the temporal structures. An alternative and more expensive approach would be to fit the complex exponentials to the time evolution of the RBF coefficients. 7 Conclusions and Outlook That was a long journey. These notes began with traditional tools for computing statistics and modal decompositions of gridded data and explored novel methods for scattered data. As fluid dynamicists, we are primarily concerned with first- and second-order statistics, used to relate random variables in vastly different contexts: the components of a vector field at a given location or at various times and locations. The covariance matrices obtained in these contexts carry different physical meanings. The covariance matrix built from velocity field components at a given location corresponds to the Reynolds stress, whose eigendecomposition provides insights into the nature of turbulence. The covariance functions (or matrices) constructed from random fields in space and time reveal coherent patterns in the data, with their eigendecomposition leading to the Proper Orthogonal Decomposition. The main challenge with scattered data from tracking velocimetry is not the lack of a grid itself but the changing sampling locations across snapshots, which complicates the evaluation of the ensemble operator. Radial Basis Functions (RBFs) provide an effective way to manage these datasets and compute the necessary integrals. They also make it possible to add physics constraints and generate analytic fields for both instantaneous snapshots and statistical fields. While the tools introduced for gridded data are well established, the meshless formalism discussed here is still a research topic, and readers are encouraged to join this research effort. The exercises and codes provided should offer a helpful starting point. Given the increasing use of tracking velocimetry in experimental fluid mechanics and the growing importance of meshless particle-based methods in computational fluid dynamics (e.g., Smoothed Particle Hydrodynamics, SPH, or Lagrangian Differencing Dynamics, LDD), meshless computations, meshless POD, and, more broadly, meshless decompositions are poised to become central themes in the near future.