Deep learning for pedestrians: backpropagation in Transformers
Laurent Boué \Hy@href mailto:[email protected] \Hy@href https://www.linkedin.com/in/laurent-bou%C3%A9-b7923853/ \Hy@href https://github.com/Ranlot \Hy@href https://twitter.com/ranlot75 · 2025
Read Deep learning for pedestrians: backpropagation in Transformers by Laurent Boué \Hy@href mailto:[email protected] \Hy@href https://www.linkedin.com/in/laurent-bou%C3%A9-b7923853/ \Hy@href https://github.com/Ranlot \Hy@href https://twitter.com/ranlot75 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23329.txt.
Introduction
Laurent Boué writes a pedagogical document that derives backpropagation for transformer networks entirely by hand. It follows his earlier vectorized treatment of convolutional networks, keeping the same principles and notation, now aimed at next-token-prediction architectures. New layer types get the full treatment: embeddings, multi-headed self-attention, and layer normalization, plus gradient formulas for LoRA adapters as an example of parameter-efficient fine-tuning.
The motivation is understanding rather than speed. Automatic differentiation will happily compute gradients for code its author does not understand, but gaps in grasping the forward pass surface the moment one tries to differentiate the loss manually. Working the backward pass by hand builds intuition for how each operation steers the final output. A complete PyTorch implementation of a minimal GPT-like network accompanies the derivations, with analytic expressions for every gradient update, so readers can check the algebra against running code.
Catalog record
| CREATOR | Laurent Boué \Hy@href mailto:[email protected] \Hy@href https://www.linkedin.com/in/laurent-bou%C3%A9-b7923853/ \Hy@href https://github.com/Ranlot \Hy@href https://twitter.com/ranlot75 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 166 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-23329
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep learning for pedestrians: backpropagation in Transformers Thanks: (work initiated while at Microsoft) Laurent Boué \Hy@href mailto:[email protected] \Hy@href https://www.linkedin.com/in/laurent-bou%C3%A9-b7923853/ \Hy@href https://github.com/Ranlot \Hy@href https://twitter.com/ranlot75 Abstract This document is a follow-up to our previous paper dedicated to a vectorized derivation of backpropagation in CNNs [ 1 ] . Following the same principles and notations already put in place there, we now focus on transformer-based next-token-prediction architectures. To this end, we apply our lightweight index-free methodology to new types of layers such as embedding, multi-headed self-attention and layer normalization. In addition, we also provide gradient expressions for LoRA layers to illustrate parameter-efficient fine-tuning. Why bother doing manual backpropagation when there are so many tools that do this automatically? Any gap in understanding of how values propagate forward will become evident when attempting to differentiate the loss function. By working through the backward pass manually, we gain a deeper intuition for how each operation influences the final output. A complete PyTorch implementation of a minimalistic GPT-like network is also provided along with analytical expressions for of all of its gradient updates. 1 Sequence modeling from 20,000 feet… Data representation. In the following, a token is understood to be any atomic unit of information such as words in natural language, pixels in an image, amino acids in proteins, time stamps in time series forecasting… A “sample” of data is understood to be a sequence of n_{\mathcal{T}} tokens where the relative arrangement of tokens with respect to each other encodes a meaningful higher-level structure. For instance, in natural language, the meaning of a sentence emerges from the way sequences of words are combined with each other to convey higher-level ideas. Similarly, in computer vision, while each pixel in an image holds a value (such as color or intensity), higher-level concepts like objects emerge when coherent sequences of pixels are considered together. Each token t is identified — via a specialized “tokenizer” — as an integer t\sim\mathbb{N}\in[1,\cdots,n_{\text{vocab}}] where n_{\text{vocab}} corresponds to the maximum number of tokens in our “vocabulary”. We refer the reader to [ 2 ] for a review of tokenizers for different data modalities. Therefore, one single data input is denoted by a vector of integers [t_{1}\sim\mathbb{N},\cdots,t_{n_{\mathcal{T}}}\sim\mathbb{N}]\sim\mathbb{N}^{n_{\mathcal{T}}} of size n_{\mathcal{T}} corresponding to the number of tokens in the sequence. Next-token prediction. Let us denote the model as a parametrized function \mathcal{N}_{\mathcal{P}} that takes as input a sequence of tokens {\bf a}_{0}\sim\mathbb{N}^{n_{\mathcal{T}}} and returns a new sequence {\bf y}_{\text{pred}}=\mathcal{N}_{\mathcal{P}}({\bf a}_{0})\sim\mathbb{R}^{n_{\mathcal{T}}\times n_{\text{vocab}}} where each token \sim\mathbb{N} is transformed into a normalized probability density vector \sim\mathbb{R}^{n_{\text{vocab}}} over the vocabulary of tokens. Alongside this probabilistic prediction, each token is also associated with a ground-truth target token which, ideally, should match as best as possible the prediction vector produced by \mathcal{N}_{\mathcal{P}} . Graphically, this can be represented as {\bf a}_{0}=\left(\begin{array}[]{ccc}t_{1}\sim\mathbb{N}\\ \vdots\\ t_{n_{\mathcal{T}}}\sim\mathbb{N}\end{array}\right)\,\,\longrightarrow\,\,{\bf y}_{\text{pred}}=\left(\begin{array}[]{ccc}{\bf y}_{\text{pred}}(t=1)\sim\mathbb{R}^{n_{\text{vocab}}}\\ \vdots\\ {\bf y}_{\text{pred}}(t=n_{\mathcal{T}})\sim\mathbb{R}^{n_{\text{vocab}}}\end{array}\right)\quad\text{vs.}\quad{\bf y}_{\text{gt}}=\left(\begin{array}[]{ccc}y_{\text{gt}}(t=1)\sim\mathbb{N}\\ \vdots\\ y_{\text{gt}}(t=n_{\mathcal{T}})\sim\mathbb{N}\end{array}\right) As usual in classification settings, the mismatch between the prediction and the ground-truth for token t is quantified via the cross-entropy loss function {\bf\ell}{P}\left({\bf y}_{\text{gt}}\,,{\bf y}_{\text{pred}}\right)=\left(\begin{matrix}-\,{\bf y}_{\text{gt}}(t=1)_{\text{OHE}}\cdot\log\,{\bf y}_{\text{pred}}(t=1)\sim\mathbb{R}\\ \vdots\\ -\,{\bf y}_{\text{gt}}(t=n_{\mathcal{T}})_{\text{OHE}}\cdot\log\,{\bf y}_{\text{pred}}(t=n_{\mathcal{T}})\sim\mathbb{R}\end{matrix}\right)=-{\bf y}_{\text{gt}}\ominus\log{\bf y}_{\text{pred}}\sim\mathbb{R}^{n_{\mathcal{T}}} applied independently to all n_{\mathcal{T}} tokens in the sequence and where the ground-truth tokens have been lifted from integers into their equivalent “One Hot Encoded” (OHE) representations. For next-token prediction tasks (relevant for model pre-training and supervised fine-tuning — SFT) the ground-truth {\bf y}_{\text{gt}} is chosen as a “shifted-by-one” version of the input {\bf a}_{0} . That is to say that, considering a token t^{\star} from the input {\bf a}_{0} , its target should be y_{\text{gt}}(t=t^{\star})=t^{\star}+1 taken from the same {\bf a}_{0} . This may be understood graphically as {\bf y}_{\text{gt}}=\text{shiftByOne}({\bf a}_{0})\quad\Longleftrightarrow\quad\hbox to298.76pt{\vbox to77.73pt{\pgfpicture\makeatletter\hbox{\hskip 14.06218pt\lower-38.865pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke} \lxSVG@begingroup@{fill} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\nullfont\lxSVG@begingroup@{_scopebegin} \lxSVG@closescope \hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \par{{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-10.72917pt}{-0.79167pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} \par{{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{190.24223pt}{-1.47221pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} \par{}{}{ {}{}{}}{{{{}{}{}}}}{{}}{{\lx@inpgf@ignorespaces}}{{}}{{}}{ {}}{{}}\lx@inpgf@ignorespaces\hbox{\hbox{\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{}{{{}}{{}}{{}}}{{{\lx@inpgf@ignorespaces}}}{{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{34.86684pt}{-29.3601pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}}{{{\lx@inpgf@ignorespaces{}}}{{}}{{}}{{}}}} \lxSVG@closescope }}}\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{}{{ }}{ } {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{24.66714pt}{0.0pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}}\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{158.38416pt}{0.0pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} \par{}{}{{{ {}{}{}}}}{{}}{{}}{{}}{ {}}{{}}\lx@inpgf@ignorespaces\hbox{\hbox{\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{}{{{}}{{}}{{}}{{}}}{{{\lx@inpgf@ignorespaces}}}{{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{228.20416pt}{-27.33234pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}}{{{\lx@inpgf@ignorespaces{}}}{{}}{{}}{{}}{{}}}} \lxSVG@closescope }}}\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{}{{ }}{ } {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{218.00447pt}{0.0pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}}\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{}{{}}{} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{278.02715pt}{0.0pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} \par \lx@inpgf@ignorespaces {{}{}{}}{}{{}}{}{ {}{}{}} {}{}\lxSVG@begingroup@{_scopebegin} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\lxSVG@setdash{3.0pt,3.0pt}{0.0pt}\lxSVG@begingroup@{stroke-dasharray,stroke-dashoffset} \lx@inpgf@ignorespaces{}{{ {\lxSVG@setlinewidth{\the\pgflinewidth}\lx@inpgf@ignorespaces\lxSVG@setdash{0.0pt}\lx@inpgf@ignorespaces\lxSVG@roundcap\lx@inpgf@ignorespaces\lxSVG@roundjoin\lx@inpgf@ignorespaces{} {}{}{} {}{}{} \lxSVG@stroke}} }{}{}{{}}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 198.9 28.69 L 318.84 9.61}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{-0.98758}{0.15712}{-0.15712}{-0.98758}{143.74477pt}{20.735pt}\lxSVG@begingroup@{transform} \lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke-width} \lxSVG@begingroup@{stroke-dasharray,stroke-dashoffset} \lxSVG@begingroup@{stroke-linecap} \lxSVG@begingroup@{stroke-linejoin} \lxSVG@drawpath@unclipped{M -2.16 2.88 C -1.98 1.8 0 0.18 0.54 0 C 0 -0.18 -1.98 -1.8 -2.16 -2.88}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}{{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{0.98758}{-0.15712}{0.15712}{0.98758}{230.42744pt}{6.94456pt}\lxSVG@begingroup@{transform} \lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke-width} \lxSVG@begingroup@{stroke-dasharray,stroke-dashoffset} \lxSVG@begingroup@{stroke-linecap} \lxSVG@begingroup@{stroke-linejoin} \lxSVG@drawpath@unclipped{M -2.16 2.88 C -1.98 1.8 0 0.18 0.54 0 C 0 -0.18 -1.98 -1.8 -2.16 -2.88}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope \lx@inpgf@ignorespaces {{}{}{}}{}{{}}{}{ {}{}{}} {}{}\lxSVG@begingroup@{_scopebegin} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\lxSVG@setdash{3.0pt,3.0pt}{0.0pt}\lxSVG@begingroup@{stroke-dasharray,stroke-dashoffset} \lx@inpgf@ignorespaces{}{}{}{}{{}}{}{}{}{{}}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 215.51 -20.71 L 314.52 -37.65}{fill:none} {{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{-0.98567}{0.1687}{-0.1687}{-0.98567}{155.75098pt}{-14.96632pt}\lxSVG@begingroup@{transform} \lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke-width} \lxSVG@begingroup@{stroke-dasharray,stroke-dashoffset} \lxSVG@begingroup@{stroke-linecap} \lxSVG@begingroup@{stroke-linejoin} \lxSVG@drawpath@unclipped{M -2.16 2.88 C -1.98 1.8 0 0.18 0.54 0 C 0 -0.18 -1.98 -1.8 -2.16 -2.88}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}{{}{{}}{}{}{{}}{{{\lx@inpgf@ignorespaces}}{{{\lx@inpgf@ignorespaces}}{\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{0.98567}{-0.1687}{0.1687}{0.98567}{227.30434pt}{-27.21257pt}\lxSVG@begingroup@{transform} \lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke-width} \lxSVG@begingroup@{stroke-dasharray,stroke-dashoffset} \lxSVG@begingroup@{stroke-linecap} \lxSVG@begingroup@{stroke-linejoin} \lxSVG@drawpath@unclipped{M -2.16 2.88 C -1.98 1.8 0 0.18 0.54 0 C 0 -0.18 -1.98 -1.8 -2.16 -2.88}{fill:none} \lxSVG@closescope \lxSVG@closescope }}{{\lx@inpgf@ignorespaces}}}}\lx@inpgf@ignorespaces \lxSVG@closescope \par \lxSVG@closescope {{ {}{}{}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}} Due to this shift-by-one property between the tokens of {\bf y}_{\text{gt}} and {\bf a}_{0} in standard next-token prediction tasks, the very first token t_{1} in {\bf a}_{0} is not used in the loss function and the number of elements contributing to the loss for a sequence of n_{\mathcal{T}} tokens is reduced to n_{\mathcal{T}}-1 . 2 Embedding layer The purpose of embedding layers is to transform categorical variables from their discrete representations — where they exist as static structureless elements of a set — into continuous and “meaningful” d -dimensional vector-based representations \sim\mathbb{R}^{d} known as “embeddings”. Here, “meaningful” implies that the learned embedding vectors are expected to capture useful and objective-dependent information (as enforced by the loss function). While tokens are the primary categorical variables for which “token embeddings” are constructed, their relative positions within a sequence also constitute an abstract categorical variable giving rise to “positional embeddings.” In the case of token embeddings, some desirable properties would, for example, be that tokens with similar semantic meanings would be associated with vector representations that are close to each other. Conversely, one would expect positional embeddings to reflect order-sensitive information. Let us consider a specific token t^{\star} from an input sequence {\bf a}_{i-1} of n_{\mathcal{T}} tokens . As a discrete unit of information, this token can always be represented as an integer {\bf a}_{i-1}(t=t^{\star})\sim\mathbb{N} but the range of possible values {\bf a}_{i-1}(t=t^{\star})\in[1,\cdots,n_{\mathcal{V}}] depends on whether we look at it from its vocabulary-membership perspective or from its position in the sequence. • For the purposes of token embeddings, the tokenizer assigns t^{\star} with {\bf a}_{i-1}^{\text{token}}(t=t^{\star})\in[1,\cdots,n_{\text{vocab}}] where n_{\text{vocab}} denotes the total number of tokens in the vocabulary. The exact integer value serves only to distinguish t^{\star} from all the other tokens in the vocabulary but carries no inherent meaning. • On the other hand, for positional embeddings, the integer representation of t^{\star} would correspond to the index of its position in the sequence of length n_{\mathcal{T}} . Therefore {\bf a}_{i-1}^{\text{position}}(t=t^{\star})\in[1,\cdots,n_{\text{context}}] where n_{\text{context}} is the context length of the model, i.e. the maximum number of tokens that the model can handle as a single sequence. Unless the sequence {\bf a}_{i-1} has been padded