Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gaps
Vasileios Sevetlidis*, George Pavlidis · 2026
Read Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gaps by Vasileios Sevetlidis*, George Pavlidis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21624.txt.
Introduction
Vasileios Sevetlidis and George Pavlidis survey the hidden memory inside deep learning training. Training is often treated as if each update stood alone. In truth, updates depend on optimizer state, data order, and a trail of auxiliary records.
The survey sorts these memory mechanisms by source, lifetime, and how visible they are. Teacher averages, contrastive queues, and normalization statistics all carry traces of the past. Staged augmentation and replay policies do too. Mapping this memory matters because it shapes results, reproducibility, and even privacy, yet it is rarely measured in a unified way.
Catalog record
| CREATOR | Vasileios Sevetlidis*, George Pavlidis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21624
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gaps Vasileios Sevetlidis*, George Pavlidis Abstract Modern deep-learning training is not memoryless. Updates depend on optimizer moments and averaging, data-order policies (random reshuffling vs with-replacement, staged augmentations and replay), the nonconvex path, and auxiliary state (teacher EMA/SWA, contrastive queues, BatchNorm statistics). This survey organizes mechanisms by source, lifetime, and visibility. It introduces seed-paired, function-space causal estimands; portable perturbation primitives (carry/reset of momentum/Adam/EMA/BN, order-window swaps, queue/teacher tweaks); and a reporting checklist with audit artifacts (order hashes, buffer/BN checksums, RNG contracts). The conclusion is a protocol for portable, causal, uncertainty-aware measurement that attributes how much training history matters across models, data, and regimes. K eywords Training memory \cdot Optimizer state \cdot Sampler state \cdot Data ordering \cdot Path dependence \cdot Causal interventions \cdot Calibration \cdot Reproducibility 1 Introduction Training a deep neural network is not memoryless. By training memory we mean that what the learner does next depends not only on its current parameters and the current minibatch, but also on how it arrived there —the recent sequence of updates and data. Specially, training memory can arise from (i) optimizer state (e.g., momentum buffers or adaptive moments that carry summaries of past gradients), (ii) sampler state (e.g., the order in which examples are presented or priorities that make some examples more likely to be seen), and (iii) the parameter path itself (the route through the loss landscape) which, in nonconvex problems, makes the order of small updates matter. Classical and modern optimization analyses formalize these ideas. Momentum and averaging explicitly carry information forward across steps, and in nonconvex settings stochastic updates do not commute—applying update A then B can lead to a different point than applying B then A —so outcomes can depend on path and ordering ( 6 ; 102 ; 85 ; 75 ; 32 ) . In practice, stateful mechanisms are used deliberately at scale: adaptive optimizers (e.g., Adam/AdamW), exponential moving averages of weights (EMA) and stochastic weight averaging (SWA), and sharpness-aware updates introduce controlled history dependence to stabilize or improve generalization ( 41 ; 23 ) . In distributed and federated settings, server-side adaptivity further exposes the role of carried-over state across rounds ( 88 ) . Memory also lives in the data pipeline. Even without explicit priorities, random reshuffling (processing each example once per epoch in a random order) behaves differently from with-replacement sampling; theory and experiments show distinct convergence properties and, at times, performance ( 94 ; 32 ; 75 ) . When sampling is stateful —for example via online or prioritized selection that revisits high-loss or high-influence examples—training can speed up, but attribution becomes entangled between data order and optimizer history ( 68 ; 48 ; 43 ; 92 ) . Beyond loss curves, studies of representation similarity (e.g., SVCCA and CKA) document that learned features can drift across runs or phases, even when top-line metrics look similar ( 86 ; 59 ; 16 ) . These tools describe that change occurs, but they are not causal diagnostics: they do not attribute changes to optimizer state, sampler state, or path effects, and they are rarely paired with effect sizes and uncertainty. Despite broad recognition that DNN training has memory, the community lacks a portable , causal , and standardized way to quantify how much that memory matters across architectures, data sets, and training regimes. Typical reports emphasize final accuracy or loss and often leave protocols under-specified (e.g., whether momentum was reset between phases, the exact data-order policy, or augmentation schedules). Order- and state-driven phenomena then become difficult to reproduce or compare. Related areas reinforce this picture at longer time scales: in continual learning, forgetting and transfer depend on task order and replay ( 82 ; 15 ) ; in curriculum learning, pacing and example ordering affect stability and sample efficiency ( 99 ) . Yet across these settings there is still no consensus diagnostic that cleanly attributes effects to optimizer state, sampler memory, or parameter-path dependence. We synthesize mechanisms and evidence for training memory across optimizers, samplers, and parameter paths, and we surface protocol pitfalls that hinder attribution. This article does not propose a new algorithm or a single diagnostic. Instead, it contributes (i) a taxonomy (source–lifetime–visibility), (ii) a synthesis of theory and evidence, (iii) causal estimands with portable perturbation primitives, and (iv) a reporting checklist that make attribution auditable. We then articulate solution-agnostic desiderata for future diagnostics: isolate sources via controlled perturbations; report effect sizes in function space alongside standard metrics; track representation drift with appropriate caveats; and emphasize early indicators that predict late generalization. Our aim is to motivate principled, causal measurement of training memory—without committing to a specific methodology in this study. 2 Relation to Prior Surveys Several survey families intersect with what we term training memory —persistent traces of optimization history, algorithmic state, and data ordering that shape the final solution. Geometry-oriented overviews on loss landscapes and model merging illuminate path dependence and mergeability (e.g., linear/curvilinear connectivity and permutation-aware alignment), but typically abstract away the concrete sources of state accumulated during training ( 22 ; 64 ; 113 ) . Work on sampler order shows that random reshuffling (without replacement) can materially alter convergence and final solutions relative to with-replacement SGD, yet these analyses are not usually framed as a unifying “memory” mechanism nor tied to broader reporting practice ( 75 ; 91 ; 115 ) . Reviews of normalization emphasize that running statistics and batch coupling are consequential sources of hidden state—with recent evidence of task-specific pitfalls—again treated largely in isolation from other mechanisms ( 39 ; 89 ) . In self-supervised and contrastive learning, surveys document explicit memory structures such as queues and banks (e.g., MoCo-style dictionaries), but scope is confined to SSL rather than integrated with optimizer/geometry/order effects ( 106 ) . Complementary lines synthesize representational and functional similarity tools for auditing training trajectories ( 54 ) , ensembling surveys that touch SWA/SWAG as temporal averaging along a path ( 114 ) , federated learning surveys that catalog server/client momentum as cross-round state ( 9 ) , and calibration/uncertainty surveys that focus on reliability outcomes rather than the upstream memory mechanisms that might drive them ( 108 ) . Documentation artifacts such as Model Cards, Datasheets, and NeurIPS reproducibility checklists advance general reporting, but they do not yet target memory-sensitive details (e.g., sampler semantics, EMA/SWA configuration, BN/SyncBN regimes, queue refresh rules) ( 76 ; 26 ; 84 ) . Our survey differs by (i) unifying these strands under a single taxonomy of training memory that treats optimizer momentum/EMA/SWA, sampler order/reshuffling, normalization running statistics, explicit buffers (queues/replay), geometric path dependence, and federated server/client state as first-class, interacting carriers of history; (ii) mapping cross-mechanism interactions (e.g., how order shapes EMA trajectories or how BN state mediates mergeability); and (iii) proposing measurement protocols and reporting checklists specific to memory. Concretely, we standardize trajectory-level similarity profiling, interpolation/merging and barrier tests, order-sensitivity ablations, state ablations for EMA/SWA/BN/queues, and calibration under controlled memory manipulations, alongside a minimal set of fields for declaring and stress-testing memory effects. This integrated perspective complements existing surveys by making the memory mechanisms explicit, comparable, and reportable across training pipelines. 3 Taxonomy of Training Memory A learner has training memory