Time-Warping Recurrent Neural Networks for Transfer Learning Abstract \authorLast Hirschi \authorFirst Jonathon \authorMiddle Michael \education B.S., University of British Columbia, 2016 M.S., University Colorado Denver, 2024 \school University of Colorado Denver \program Applied Mathematics \reportType dissertation \degreeName Doctor of Philosophy \submitDay 22 \submitMonth April \submitYear 2026 \advisor Jan Mandel \advisorTitle Dr. \committeeChair Joshua French \committeeMembers Kyle Hilburn, Angel Farguell, Farhad Pourkamali \ucdabstract Dynamical systems describe how a physical system evolves over time. Physical processes can evolve faster or slower in different environmental conditions. We use time-warping as rescaling the time in a model of a physical system. This thesis proposes a new method of transfer learning for Recurrent Neural Networks (RNNs) based on time-warping. We prove that for a class of linear, first-order differential equations known as time lag models, an LSTM can approximate these systems with any desired accuracy, and the model can be time-warped while maintaining the approximation accuracy. The Time-Warping method of transfer learning is then evaluated in an applied problem on predicting fuel moisture content (FMC), an important concept in wildfire modeling. An RNN with LSTM recurrent layers is pretrained on fuels with a characteristic time scale of 10 hours, where there are large quantities of data available for training. The RNN is then modified with transfer learning to generate predictions for fuels with characteristic time scales of 1 hour, 100 hours, and 1000 hours. The Time-Warping method is evaluated against several known methods of transfer learning. The Time-Warping method produces predictions with an accuracy level comparable to the established methods, despite modifying only a small fraction of the parameters that the other methods modify. \advisorapprovalpage true \copyrightpage true Acknowledgements. The research presented in this paper was partially funded by NASA grants 80NSSC22K1717, 80NSSC23K1118, 80NSSC22K1405, 80NSSC23K1344, and 80NSSC25K7276. Special thanks to my research collaborators at the Wildfire Interdisciplinary Research Center (WIRC) at San Jose State University, and the Cooperative Institute for Research in the Atmosphere (CIRA) at Colorado State University. Computing resources were provided by the University of Colorado Denver supercomputing cluster Alderaan, partially funded by NSF Campus Cyberinfrastructure grant OAC-2019089. A portion of this work used code generously provided by Brian Blaylock’s Herbie Python packages Herbie (Version 2024.8.0) ( https://github.com/blaylockbk/SynopticPy ) and SynopticPy ( https://github.com/blaylockbk/SynopticPy ). Large Language Models, specifically ChatGPT-5, were utilized for support with coding, literature search, and writing revisions. Chapter 1 INTRODUCTION Many physical systems change over time, and these changes often follow patterns that reflect how quickly the system responds to new conditions. The characteristic time scale of a system refers to the typical amount of time over which the system responds to changes in its conditions. It is often useful from a modeling perspective to modify the time scale to represent different dynamics. In this thesis, we refer to the modification of the time scale of the dynamics as “time-warping". Time-warping a physical system is used for the purposes of empirically fitting to data, where the observed rate of change of a system is different from what the model predicts and you need to adjust the dynamics to match reality. Time-warping is also useful when modeling a related system. It is common in applied time series modeling to assume that two different systems are described by the same underlying dynamics, with only the difference that one system is faster or slower than the other. Mathematical models attempt to identify dynamics that are invariant to different temporal changes. In a machine learning (ML) framework, transfer learning refers to a set of methods where aspects of one problem are applied to a related problem. Time-warping a dynamic system can be framed as a form of transfer learning. Traditional differential equations model the rate of change of a system. The dynamics can be directly sped-up or slowed-down. Modern ML methods for sequence learning typically learn a discrete update rule of the system. Recurrent neural networks (RNNs), including the popular Long Short-Term Memory (LSTM), are trained on a certain discrete time scale. These models are often viewed as black-boxes, and the characteristic time scale of the system is not obvious and not amendable to modification. In this thesis, we will show how the learned dynamics of an RNN can be time-warped. Under a certain set of assumptions about the underlying system and the RNN that represents it, this can be proven mathematically. For a class of first order linear differential equations known as time lag models, we prove that an LSTM can approximate the systems with arbitrary accuracy. Then for type of temporal rescaling of the dynamic system that we define as a time-warp, we prove that specific parameters in the LSTM can be modified to approximate the time-warped system. The proofs provide the theoretical foundation of a practical method for time-warping a pretrained RNN. The time-warping method will be demonstrated in a simulation with idealized dynamics. Then, the time-warping method will be evaluated statistically in an applied transfer learning problem. The motivating application of this thesis is to improve the accuracy of modeling fuel moisture content (FMC) for fuels that have sparse observations. FMC is an important concept in wildfire science that affects wildfire risk and active wildfire behavior. Fuels of different sizes respond to environmental conditions differently, and the FMC evolution over time is faster or slower depending on the size of the fuel particles. Large quantities of data exist for fuels with a characteristic response time of 10 hours (10h fuels), but the observations for fuels with other response times are sparse. Fuels with characteristic response times of 1h, 100h, and 1000h are studied by researchers and used in operational wildfire modeling tools, but observations of these fuels are low resolution in both space and time. In this thesis, we will use an RNN with LSTM recurrent layers that was pretrained on observations from 10h fuels. We will apply a time-warping technique to generate FMC predictions for 1h, 100h, and 1000h fuels, then evaluate this method against known transfer learning techniques. The pretrained RNN with LSTM recurrent layers was developed for 10h fuels using observations from a network of standardized sensors across the Continental United States (CONUS). The model was evaluated extensively and compared to several ML baselines [ 12 ] . A small-scale study from 1996-1997 in Oklahoma has been extensively relied upon in the FMC modeling literature for predicting the FMC of 1h, 100h, and 1000h fuels [ 5 ] . Time-warping the pretrained RNN will be evaluated and compared to established methods within transfer learning using this influential FMC data set. This thesis fills several gaps within the RNN literature. First, formally framing time-warping as a transfer learning problem has not been studied to the best of our knowledge. Approaching time-warping as a transfer learning problem helps advance the field of transfer learning with RNNs. Then, while there has been some research into how RNNs learn a characteristic time scale, to our knowledge there exists no literature on post-hoc modification of the learned dynamics of an RNN. That is, taking a pretrained RNN and attempting to modify the learned temporal dynamics is a novel concept with interesting potential applications. The time-warping framework leverages knowledge about temporal dynamics to enable more effective transfer learning. This thesis also contributes to improving the interpretability of RNN-based models. We develop a theoretical result showing that a class of dynamical systems can be approximated with an LSTM and then time-warped to represent different time scales with any desired accuracy. Directly modifying specific weights inside black-box RNNs based on theoretical considerations is rare within machine learning. Further, we use mathematical properties of the LSTM to develop theoretical expectations for how certain weights may change during transfer learning. Developing a priori expectations for how neural network parameters may respond to new observations can help build a deeper understanding of RNN behavior and clarify some aspects of their black-box nature. Lastly, this thesis advances FMC modeling by introducing a new method of transfer learning that is highly applicable to this field. In recent years, ML techniques are beginning to augment or improve upon traditional physics-based approaches to predicting FMC. Physically based models can be used to make informed predictions on how fuels will behave when there are sparse observations, while ML models tend to struggle when there are not large quantities of data to learn patterns from. Transfer learning has the potential to greatly improve FMC modeling in cases where observations are sparse. The time-warping method will be shown to be accurate compared to established transfer learning techniques, despite only modifying a small fraction of the parameters that the other techniques modify. All computational results in this thesis can be reproduced using publicly available code. See Appendix 7.2 for more details. Chapter 2 BACKGROUND 2.1 Time-Warping Dynamic Systems Throughout this work, the term “time series” refers to discrete-time models and data. The term “dynamical systems” refers to models defined by an explicit evolution operator, most commonly in continuous time. Differential equations are a subset of dynamical systems that are defined in continuous time and describe instantaneous rates of change. In practice, differential equations are discretized with respect to some temporal grid. Digital technologies like computers and measurement devices have led to the implementation of differential equations in discrete time. Many differential equations are formulated in a way that does not fix an intrinsic temporal resolution, allowing the same model to be evaluated on different time scales. A modeler can choose to run a differential equation on an hourly-resolution temporal grid or a daily-resolution grid or other resolutions, though this raises issues related to accuracy and stability. Whether the model accurately describes reality on those different scales is a separate question. Many statistical approaches to time series are defined in discrete time and describe the change in a system over a single time step. RNNs describe a discrete update to a system on a fixed temporal grid. Classical statistical approaches to time series work similarly, like autoregression (AR) and moving average methods. For statistical and ML models in discrete time, it is often intractable to deploy such models on a temporal grid with different spacing. Particularly with neural networks, which are often viewed as black boxes, there are few established methods that allow for this. Consider a general discrete-time model of the form z_{t}=f(z_{t-1},X_{t}) with initial condition z_{0} . Let N be a positive integer, and define the discrete time index t=0,1,\dots,N . The model evolves according to the update rule for t=1,\dots,N . We denote the sequence (z_{t})_{t=1}^{N} and the inputs (X_{t})_{t=1}^{N} . This indexing convention will be used throughout the thesis. A single time step might represent an hour. To run the model on daily resolution, you could generate z_{0},z_{24},z_{48},\dots by iterating the model forward 24 steps at a time. However, to evolve the system forward from the initial condition z_{0} to z_{24} , you would need to input X_{1},X_{2},\dots,X_{24} . In practice, the input data might not be measured hourly, and then there is no obvious way to translate a one-step model into a 24-step model. So, an RNN trained on daily resolution data cannot be trivially translated to a produce hourly resolution predictions. We denote a time-warp by \gamma as a mapping t\mapsto\gamma t , for some positive real number \gamma>0 . Consider a general first-order system in continuous time, \frac{dy(t)}{dt}=f\!\left(y(t),X(t)\right) where X(t) denotes external inputs. The function f also has a set of parameters that are fixed over time. Define a time-warp by \tilde{t}=\gamma t with \gamma>0 , and identify \tilde{y}(\tilde{t}):=y(t),\qquad\tilde{X}(\tilde{t}):=X(t),\quad t=\tilde{t}/\gamma. Applying the chain rule, \frac{d\tilde{y}(\tilde{t})}{d\tilde{t}}=\frac{dy(t)}{dt}\frac{dt}{d\tilde{t}}=\frac{1}{\gamma}f\!\left(\tilde{y}(\tilde{t}),\tilde{X}(\tilde{t})\right). At this point, there are two mathematically equivalent ways to use this formulation that have different interpretations. Temporal rescaling treats \tilde{t} as a new time variable with different units. If t represents hours, then \gamma=24 would mean that \tilde{t} represents days. So, temporal rescaling fixes the dynamics but changes the units of time the dynamics are evaluated on. Alternatively, if we keep t and \tilde{t} as the same time scale, and rescale the system after the time-warp, then we have modified the dynamics of the original system while keeping time fixed. The first argument of function f has been scaled by 1/\gamma , with time being unchanged. If \gamma>1 , we have sped up the system, and if \gamma<1 , we have slowed down the system. This thesis focuses on time-warping in the context of transfer learning for RNNs, where an RNN is pretrained on data at one timescale, then time-warped to model a related system that operates at a faster or slower timescale. We refer to time-warping in this thesis to distinguish it from standard temporal rescaling for this reason. 2.2 Time Lag Systems Consider a linear first-order differential equation with initial conditions, \frac{dz(t)}{dt}=\lambda(X-z(t)),\quad z(0)=z_{0},\;\lambda>0 where \lambda is a fixed positive parameter and X is constant in time. A closed-form analytical solution can be obtained by separation of variables, \displaystyle\frac{dz}{X-z} \displaystyle=\lambda\;dt \displaystyle\int\frac{1}{X-z}dz \displaystyle=\int\lambda\;dt \displaystyle-\ln|X-z| \displaystyle=\lambda t+\mathcal{C} \displaystyle z \displaystyle=X-\mathcal{C}e^{-\lambda t} where \mathcal{C} denotes a constant of integration. Using the initial condition, we have \mathcal{C}=X-z_{0} , and the analytical solution is \displaystyle z(t)=X+(z_{0}-X)e^{-\lambda t}. (2.1) The solution exhibits exponential relaxation. The state z(t) converges to the equilibrium value X , with the rate of convergence determined by the parameter \lambda . Next, we discretize the solution in time by introducing a constant sampling interval \Delta t . We allow the input to vary in time under a zero-order hold assumption, meaning that X(t) is held constant over each sampling interval. Specifically, over the interval [t_{k},t_{k}+\Delta t) , we take X(t)=X_{k} . Over this interval, the differential equation becomes \frac{dz}{dt}=\lambda(X_{k}-z),\qquad z(t_{k})=z_{k}. Since X_{k} is constant on the interval, the continuous-time solution gives z(t)=X_{k}+(z_{k}-X_{k})e^{-\lambda(t-t_{k})}. Evaluating at t_{k-1}+\Delta t=t_{k} , we obtain z_{k}=X_{k}+(z_{k-1}-X_{k})e^{-\lambda\Delta t}. Rewriting, z_{k}=(1-e^{-\lambda\Delta t})X_{k}+e^{-\lambda\Delta t}z_{k-1}. For \Delta t=1 , this simplifies to z_{k}=(1-e^{-\lambda})X_{k}+e^{-\lambda}z_{k-1}. (2.2) Note that for \lambda>0 , e^{-\lambda}\in(0,1) . Define a:=e^{-\lambda} . In this thesis, we define a time lag system as a class of discrete-time linear models with the recurrence relation \displaystyle z_{t}=az_{t-1}+(1-a)X_{t},\quad 00. (2.4) 2.3 Sequence Learning with Recurrent Neural Networks Sequence learning is an area of ML that deals with ordered sequences of observations. This includes time series modeling, where observations are ordered in time, and other areas that are not time-ordered. Examples of sequence learning problems that are not time-ordered include natural language processing (NLP) and DNA sequencing. There are some important theoretical differences between time series and more general sequence modeling tasks. In time series modeling, there is a notion of causality in the dynamics. Observations from the past can influence observations in the future, but not vice-versa. In NLP and DNA sequencing, the sequences do not have an intrinsic direction of causality. In NLP, for instance, word order is often a subjective stylistic choice, and the conjugation of a word can be influenced by words that occur after it in a sequence of text. Many of the same statistical techniques are used throughout the different sub-fields of sequence modeling, but in specific applications some techniques are more appropriate. This thesis will consider time series modeling, but many of the principles will extend to general sequence modeling. Recurrent neural networks (RNNs) are a class of neural network architectures developed for sequence modeling tasks. They build upon the architecture of classical neural networks. The classic neural network unit is the perceptron, but in modern practice it is referred to as the “dense” or fully-connected unit. Multiple dense units are stacked into a dense layer. It has connections to the previous layer, possibly the input layer, that result in a linear combination that is passed through an activation function. The activation function is typically nonlinear. In modern neural network practice, a sensible default activation function for dense layers is the Rectified Linear Unit (ReLU) function, which is defined \text{relu}(x)=\max(0,x). The output of a dense layer feeds into the next layer of the neural network, or it is the output of the network if it is the final layer. A neural network that only consists of dense layers is referred to as a feed-forward network, since information flows through the input layer to the output in only one direction. This type of architecture is sometimes commonly called a multilayer perceptron (MLP). In contrast to standard feed-forward neural networks, RNNs maintain a state over time that is used to modify the output at the next time step. For many RNN architectures, this is a single value for the recurrent unit and is called the hidden state. We will refer to this as the “recurrent state" to unify the terminology for the LSTM cell, which uses a two-dimensional recurrent state. A single cell, or unit, is the basic recurrent computational unit. Collections of recurrent cells are arranged in a recurrent layer, and then multiple layers can be arranged sequentially. Typically, the recurrent layers feed into one or more dense layers to structure the output of the network. The number of units in the final output layer and the activation function of the output layer is chosen by the nature of the problem. The number of units in the output layer typically corresponds to the dimensionality of the output. In this thesis, the target output will be a continuous scalar value, so the final output layer will be a single dense cell with linear activation. If a single probability was the target output of the network, such as when predicting a binary class probability, the output layer would be a single dense cell with an activation function that mapped to the interval [0,1] . If the target output of the neural network was a real-valued vector of length two, the final output layer would be two dense cells with linear activation. Other modeling problems will lead to different choices of output layer structure. We will refer to an RNN as any neural network that contains one or more recurrent layers. There are different types of RNN architectures. The simple RNN is the most basic version, but it suffers from stability issues during training which will be discussed later in this thesis. The Long Short-Term Memory (LSTM) augments the simple RNN in complex ways and addresses many of the stability issues with training. This thesis focuses on a new method for transfer learning with LSTMs. Other RNN architectures exist, like the Gated Recurrent Unit (GRU), but they will not be treated in this thesis. 2.3.1 The Simple RNN The simple RNN augments the dense cell by adding a recurrent connection. A recurrent layer will have one or more simple RNN units. Each RNN unit has a scalar-valued recurrent state that is updated over time. The vector h_{t} represents the recurrent states for the entire recurrent layer. To produce an output at time t , a recurrent layer receives a vector of external predictors at the same time X_{t} and the vector of recurrent states at the previous time h_{t-1} . Denote the output of the layer as y_{t} . In the case of the simple RNN, the output is equal to the hidden state. But for the LSTM, it is useful to label these separately. The inputs are passed into a linear combination and then a univariate activation function \phi:\mathbb{R}\to\mathbb{R} is applied [ 10 , Ch. 15, Eq. 15–1] , \displaystyle h_{t} \displaystyle=\phi(W_{x}\,X_{t}+W_{h}\,h_{t-1}+b) (2.5) \displaystyle y_{t} \displaystyle=h_{t}, where W_{x} contains the weights connecting the inputs to the recurrent layer, W_{h} contains the recurrent weights connecting the previous states to the current states, and b is a bias vector. The vectors are treated as column vectors so the products W_{x}X_{t} and W_{h}h_{t-1} are standard matrix multiplications. The bias b is a constant contribution to the argument of the activation function. The left side of Figure 2.1 shows the computational flow of the simple RNN. There are many possible choices for the activation function in the standard software libraries [ 18 ] . The most popular choice of activation function is the hyperbolic tangent ("tanh"), which maps from the real numbers to the range (-1,1) , \displaystyle\text{tanh}(x)=\frac{e^{x}-e^{-x}}{(e^{x}+e^{-x})} (2.6) The weights and biases of the recurrent unit are kept constant during prediction and applied iteratively at each time step. For a finite sequence of length N of external inputs (X_{t})_{t=1}^{N} , and an initial recurrent state of h_{0} , the RNN generates a sequence of outputs for each time (y_{t})_{t=1}^{N} . The right side of Figure 2.1 shows the RNN “unrolled" in time. Figure 2.1 : Two complementary views of an RNN. (a) Left: a single RNN cell, following the formulation in [ 10 , p. 539] . (b) Right: the RNN unrolled in time. 2.3.2 Long Short-Term Memory The Long Short-Term Memory (LSTM) augments the simple RNN by maintaining the hidden state and an additional long-term cell state. The combination of these two states will be referred to as the recurrent state. The LSTM was originally developed by 13 to address shortcomings of the simple RNN. First, the LSTM is capable of learning longer-term relationships than the simple RNN. Second, the LSTM addresses instability in the simple RNN when calculating gradients, an issue known as the exploding and vanishing gradient problem, which will be discussed in greater detail in Section 2.3.3 . The standard LSTM equations make use of the sigmoid function, commonly denoted \sigma , \displaystyle\sigma:\mathbb{R}\to(0,1),\quad\sigma(x)=\frac{1}{1+e^{-x}} (2.7) The LSTM unit maintains two separate states over time that are interpretable as memory. The hidden state, denoted h_{t} , is interpretable as the short-term memory and is also the output of the LSTM. It is recomputed at each time step from the current input and the cell state, so it can change quickly and does not preserve past information unless it is maintained in the cell state. The cell state, denoted c_{t} , is interpretable as the long-term memory. It is incrementally modified over time. The forget gate controls how much of the previous state is retained, while the input gate controls how new information is added. The cell state persists over time and evolves gradually. The pair of these two states [c_{t},h_{t}] will be referred to as the recurrent state from now on. The recurrent state is modified through a series of computational gates. At time t , each of the computational gates receives as an input the hidden state of the recurrent layer at the previous time h_{t-1} and a vector of the external input at the same time X_{t} . The computations of an LSTM, as formulated in 10 are shown in Figure 2.2 , Figure 2.2 : A single LSTM unit. This is an original graphic, developed following the formulation in [ 10 , p. 569] . The computations are described in Equations ( 2.8 ). \displaystyle f_{t} \displaystyle=\sigma(W_{xf}\,X_{t}+W_{hf}\,h_{t-1}+b_{f}) (2.8) \displaystyle i_{t} \displaystyle=\sigma(W_{xi}\,X_{t}+W_{hi}\,h_{t-1}+b_{i}) \displaystyle g_{t} \displaystyle=\tanh(W_{xg}\,X_{t}+W_{hg}\,h_{t-1}+b_{g}) \displaystyle o_{t} \displaystyle=\sigma(W_{xo}\,X_{t}+W_{ho}\,h_{t-1}+b_{o}) \displaystyle c_{t} \displaystyle=f_{t}\otimes c_{t-1}+i_{t}\otimes g_{t} \displaystyle h_{t} \displaystyle=o_{t}\otimes\tanh(c_{t}) Again, the products are standard matrix multiplications. The symbol “ \otimes " is the Hadamard product, or element-wise multiplication. The trainable weights are described as follows. W_{xf},W_{xi},W_{xg} , and W_{xo} are weight matrices that connect the set of external inputs to each gate. W_{hf},W_{hi},W_{hg} , and W_{ho} are weight matrices that connect each gate to the hidden state of the LSTM layer at the previous time step h_{t-1} . Lastly, b_{f},b_{i},b_{g} , and b_{o} are the vectors of bias terms for each gate. The recurrent state of an LSTM cell consists of two components, the hidden state and the cell state, formulated as the matrix [h_{t},c_{t}] . The computational flow over time in Figure 2.1 still applies with an LSTM as the recurrent layer. The full recurrent state is maintained over time, but only the hidden state h_{t} is passed to downstream layers in the neural network. In other words, the cell state c_{t} is not produced as an output of the recurrent layer. The gates can be interpreted heuristically as follows. The forget gate f_{t} controls how much of the previous cell state to retain. A larger forget gate activation increases the fraction of the previous cell state that is retained, so the network retains more past information and therefore “forgets” less. Conversely, a smaller forget gate activation reduces the retained fraction of the previous cell state, causing the network to retain less information from the past. The input gate i_{t} controls how much information from the current input to add to the cell state. The candidate memory g_{t} represents a bounded nonlinear transformation of the current input and previous hidden state, and serves as the proposed update to the cell state. The output gate o_{t} modulates the extent to which the cell state contributes to the hidden state. The recurrence of c_{t} is linear, consisting of multiplication by f_{t} , which lies in (0, 1), and addition of the candidate memory times the input gate, which lies in (-1, 1). This leads to more stable dynamics and slower evolution over time, rather than repeatedly applying full linear transformations and nonlinear activation functions. For an LSTM layer with N units, each unit retains its own cell state which is not directly shared with the other units. However, each gate receives the full hidden state h_{t-1} at time t , which contains information from the cell state of each unit. So for an RNN with N>1 , the cell states c_{t} are coupled to each other. Figure 2.2 displays the LSTM architecture for a single unit using the standard activation functions, with three sigmoid activations and two tanh. However, software implementations of the LSTM allow for different activations functions to be used, including Keras [ 17 ] . 2.3.3 Optimization with Backpropagation through Time The parameters of neural networks are learned through a procedure that attempts to minimize a loss function. The loss with respect to model parameters for complicated models like neural networks cannot be assumed to be convex, and finding analytical solutions to minimize the loss is infeasible in all but the most contrived examples. The standard approach is a method known as gradient descent, which iteratively modifies model parameters by moving in the opposite direction of the gradient of the loss with respect to model parameters, e.g. [ 10 , Ch. 4] . Calculating the gradient of the loss across the entire training dataset tends to lead to the algorithm getting stuck in local minima. Batch gradient descent is an augmentation that groups the training samples into batches and calculates gradients with respect to a subset of the overall training data. By randomizing the order of samples used in the batching, stochasticity is introduced to the optimization algorithm. In practice, adding randomness to the parameter search helps to avoid local minima and leads to better solutions. Backpropagation is an efficient algorithm for calculating the gradient of a loss with respect to model parameters through successive applications of the chain rule [ 11 , Sect. 6.5] . The algorithm involves a forward pass, where inputs flow through the neural network architecture to generate an output. A loss is calculated by comparing the output of the neural network to a known target value for a specified loss function. The backward pass of the algorithm calculates the gradient of the loss with respect to each model parameter by applying the chain rule of calculus. Finally, the trainable parameters of the neural network are updated by stepping in the opposite direction of the gradient of the loss with respect to the model parameters. For RNNs, the model parameters are shared across time while the recurrent state evolves recursively to generate outputs. The gradient of the loss with respect to the model parameters must account for this temporal dependence, a modification known as Backpropagation Through Time (BPTT) [ 47 , Ch. 9.7] . Optimization via gradient descent requires computing gradients with respect to these parameters, including the weights and biases associated with recurrent connections. As an intermediate step, this involves computing gradients of the loss with respect to the recurrent state. The recurrent state h_{t} depends on all previous states h_{t-1},h_{t-2},\dots,h_{0} , where h_{0} is the initial state. Repeated application of the chain rule leads to a product of terms: for a single-unit recurrent layer, this product consists of partial derivatives with respect to h_{t-1},h_{t-2},\dots,h_{0} , while for multi-unit layers it consists of Jacobians of the state transitions. Computing this product is computationally linear in sequence length but highly unstable. In practice, gradients may grow or decay exponentially, leading to the exploding and vanishing gradient problem. The most common way to address the exploding and vanishing gradient problem is to approximate the full derivatives by truncating the recursive application of the chain rule. Shorter sequences avoid the numerical issues of exploding and vanishing gradients. This introduces a hyperparameter known as sequence length that needs to be tuned. This method is known as Truncated Backpropagation through time (TBTT) [ 47 , Sect. 9.7] . Combining TBTT and batch gradient descent leads to a three-dimensional structure for the training data for RNNs. Training samples are arranged in a matrix of the form (batch size, sequence length, features) , where features are the number of unique inputs to the model. A single sample is a time-ordered matrix of the shape (sequence length, features) . The sample passes through the RNN layers to generate an output, which is compared to a target vector of observed data to calculate a loss. It is a modeling choice whether all the outputs of length equal to the sequence length is used to calculate a loss, or just the last element in the sequence. In the Keras and TensorFlow set of software tools, the function argument return_sequences controls this behavior [ 17 ] . When return_sequences is true, the loss calculation for a single sample is the loss averaged over the entire sequence length window. The algorithm proceeds by calculating losses for each sample in a batch, then aggregating the loss calculations, typically through averaging [ 16 ] , and then the model parameters are updated in the opposite direction of the gradient in a step proportional to the learning rate hyperparameter. A single epoch of training is complete when all samples in all batches have been processed. The number of epochs used for training is a hyperparameter that requires tuning. The method known as “early stopping” dynamically halts training when there are signs that the model predictive accuracy stagnates. Early stopping utilizes a validation set during training. After each epoch of training, the loss on the validation set is calculated. Training is halted once the accuracy on the validation set fails to improve after a predetermined number of epochs, which is known as the “patience” hyperparameter. Then the final set of model weights is extracted after the epoch that had the lowest loss on the validation set. In practice, you can set the number of epochs to some very large number and utilize an early stopping patience of 5 to 10, which alleviates the need to precisely tune the number of epochs. It is still possible that the loss would have continued to decrease after stagnating for 5 to 10 epochs, so early stopping does not guarantee that the resulting model is as accurate as possible without overfitting. Training RNNs is notoriously tricky. The solution can be highly sensitive to hyperparameter choices. Though many methods exist to mitigate the issues, the exploding and vanishing gradient problem remains and can lead to computational errors or slow convergence. The sequential way that RNNs generate predictions makes RNNs less amenable to parallelization as other neural network methods. For these reasons, modern NLP has largely moved away from RNNs in favor of attention mechanisms and transformer architectures. We will briefly discuss why RNNs are still useful for applied modeling of physical systems in the next section. 2.3.4 Machine Learning Alternatives to RNNs Convolutional neural networks (CNNs) have also been applied to time series analysis. While CNNs are widely used for image processing and other spatially distributed data, networks can also be constructed using one-dimensional temporal convolutions. In these architectures, convolutions act as temporal feature extractors. This approach has become increasingly common in environmental science in recent years. For example, 48 and the follow-up study by 23 use one-dimensional temporal convolutions to predict live fuel moisture content point-wise across CONUS. Attention mechanisms are an ML approach to sequence learning that differs fundamentally from recurrent neural networks (RNNs). In an attention layer, each element of the output sequence is represented as a weighted sum of linear combinations of all elements in the input sequence. The model learns the coefficients of the linear combinations, while the weights in their sum change depending on the element currently being processed. As the current element changes, the weighting of the other elements changes accordingly, which is commonly interpreted as representing the context within the sequence. In a seminal 2017 paper, researchers from Google introduced the transformer architecture, which is built around attention mechanisms for sequence learning [ 40 ] . Transformers proved highly effective for NLP tasks, and modern large language models are primarily based on the transformer architecture. Transformers are highly parallelizable in a way that RNNs are not. RNNs maintain a memory over time and generate outputs sequentially, which limits opportunities for parallel computation. One of the main motivations for the transformer architecture was to model very long-term dependencies that RNNs struggle to capture. These long-range dependencies are particularly relevant in natural language processing, where context and logical coherence may need to be maintained across thousands of elements in a sequence. RNNs, by contrast, tend to gradually forget the past. This tendency proved ill-suited to NLP [ 40 ] , but it can be appropriate and even desirable for many physical systems. Placing low importance on the beginning of a conversation or document can lead to nonsensical language modeling, but for physical problems such as weather forecasting it often makes sense to place greater importance on more recent conditions than on the distant past. 2.4 Autocorrelation and Autoregression Autocorrelation and autoregression are concepts from classical time series analysis. While they are commonly used for modeling temporal processes, they can also be used to characterize the empirical temporal dependencies of an arbitrary time series. In Chapter 4 , these concepts are used to analyze the temporal dependencies present in predictions generated by RNN models. Autocorrelation is the correlation between a discrete time series and itself shifted in time. For a sequence (z_{t}) , where the index set may be finite or infinite, let k be an integer time interval. The covariance between (z_{t}) and (z_{t+k}) is called the autocovariance at “lag k ” [ 3 , Sect. 2.1] . Note that this use of the term “lag” is different from the concept of time-lag systems discussed above, but there is a mathematical relationship between these concepts that we will discuss shortly. The autocovariance at lag k is defined as, \text{cov}(z_{t},z_{t+k})=\mathbb{E}[(z_{t}-\mu)(z_{t+k}-\mu)], (2.9) where \mathbb{E} is the expected value, and \mu is the mean of the sequence. If (z_{t}) is an infinite sequence, it needs a well-defined mean and variance for the autocovariance to be defined. If (z_{t})_{t=1}^{N} is a finite sequence, then \mu can be estimated as the empirical sample mean, and the autocovariance can be estimated empirically with, \hat{\text{cov}}(z_{t},z_{t+k})=\frac{1}{N}\sum_{t=1}^{N-k}(z_{t}-\hat{\mu})(z_{t+k}-\hat{\mu}),\qquad\hat{\mu}=\frac{1}{N}\sum_{t=1}^{N}z_{t}. (2.10) The normalization uses N rather than N-k so that the same scaling is applied for all lags, which simplifies the definition of the autocorrelation function since the normalization constants cancel in the ratio [ 3 , Eq. 2.1.12] . The autocorrelation function (ACF) scales the autocovariance between -1 and 1 by normalizing by the variance of the process. The autocorrelation at lag k is denoted \rho_{k} and is defined as, \rho_{k}=\frac{\mathbb{E}[(z_{t}-\mu)(z_{t+k}-\mu)]}{\mathbb{E}[(z_{t}-\mu)^{2}]}=\frac{\text{cov}(z_{t},z_{t+k})}{\text{var}(z_{t})} (2.11) A time series with no temporal dependencies would be expected to have zero autocorrelation for all lags. The white noise process, denoted \epsilon_{t} , is a time series of i.i.d. random terms with no temporal structure, and the ACF is zero for all lags. For a finite sequence (z_{t})_{t=1}^{N} , the autocorrelation can be estimated empirically as, \hat{\rho}_{k}=\frac{\sum_{t=1}^{N-k}(z_{t}-\hat{\mu})(z_{t+k}-\hat{\mu})}{\sum_{t=1}^{N}(z_{t}-\hat{\mu})^{2}}=\frac{\hat{\text{cov}}(z_{t},z_{t+k})}{\hat{\text{var}}(z_{t})} (2.12) Autoregressive processes are discrete-time stochastic processes defined in terms of previous values of the series [ 3 , Sect. 3.2] . An autoregressive process of order p , denoted AR(p), and is defined as, z_{t}=\sum_{k=1}^{p}\beta_{k}\,z_{t-k}+\epsilon_{t}, (2.13) where \beta_{k} are fixed parameters and \epsilon_{t} is a white-noise process. For the ACF of an AR(p) process to be well-defined, the parameters must satisfy certain constraints that ensure the process is stable [ 3 , Sect. 3.2.3] . The simplest autoregressive process is the AR(1), which is defined by a recursive relationship of the form z_{t}=\beta\,z_{t-1}+\epsilon_{t} (2.14) The theoretical ACF at lag k can be derived analytically [ 3 , Eq. 3.2.12] and is given by \rho_{k}=\beta^{k},\quad k\geq 0. Under the condition |\beta|<1 , the ACF of the AR(1) process decays exponentially to zero as the lag increases. It is possible to fit the parameters of an AR(p) process to data, but properties of the AR(p) process can also be used to analyze the temporal structure of an observed time series. Autoregressive processes of different orders p have different theoretical forms for their ACF. By comparing the empirical ACF of an observed time series to the theoretical ACF of an AR(p) process, conclusions can be drawn about the underlying temporal dependencies. The plot of lag k versus the ACF at lag k is an important diagnostic tool for analyzing temporal dependencies in a time series. The partial autocorrelation function (PACF) measures the correlation between z_{t} and z_{t-k} after accounting for the influence of the intermediate lags 1,\dots,k-1 [ 3 , Sect. 3.2.5] . For positive \beta , the ACF of the AR(1) process begins at \rho_{0}=1 and then decreases monotonically toward zero as the lag increases. The PACF takes the value \beta at lag 1 and is zero for all lags k>1 . For a finite sample from a white-noise process with no temporal dependence, the true ACF and PACF are zero at all nonzero lags, but empirical estimates may deviate from zero due to sampling variability. In particular, the ACF at a single lag may appear nonzero purely by chance. Approximate 95 \% confidence bounds for the ACF under the white-noise assumption can be calculated and are the same for all lags. These bounds are commonly used as a statistical significance threshold to determine whether an observed ACF value at a given lag is larger than would be expected under the null hypothesis of zero temporal dependence. In diagnostic ACF plots, these approximate 95 \% bounds are typically shown as two horizontal lines. The same significance threshold is often applied to PACF plots. We previously defined the time-lag system as z_{t}=az_{t-1}+(1-a)X_{t} , with 0