Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers
Kennon Stewart \affilnum 1,2 · 2026
Read Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers by Kennon Stewart \affilnum 1,2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23046.txt.
Introduction
Stewart argues that current definitions of machine unlearning are too vague for second-order optimizers. In tests of first- and second-order learners on data deletion with varying eigendecomposition, he finds both match the ideal counterfactual in performance and gradient. The second-order optimizer, however, shows large volatility in its state, indicating hidden information that first-order analysis cannot detect. Stability and information loss return only under controlled state perturbation where geometric memory is erased.
Catalog record
| CREATOR | Kennon Stewart \affilnum 1,2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26559 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23046
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers Kennon Stewart \affilnum 1,2 Abstract We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners for their ability to handle the data deletion task with varying degrees of eigendecomposition to mimic the loss model memory. While both first and second-order methods realign with the ideal counterfactul in terms of performance and gradient, the second-order optimizer shows significant volatility in the optimizer state. This indicates residual information, supposedly deleted, that isn’t detectable by first-order analysis. Various eigendecay treatments show that stability and information loss is regained only under controlled state pertubation where geometric information (or memory) is erased. \corrauth Kennon Stewart, Second Street Labs \version v1 \preprintnumber 2026-001 \lablogo formatting/ssl/ssl.png Information is "learned" by changing the geometry of the second-order optimizer state. But what is the shape of the optimizer when forced to forget? We look at the geometry of machine unlearning. keywords machine unlearning, privacy, convex optimization 1 Introduction Machine unlearning aims to remove the influence of specific data from a trained model, often to satisfy privacy, regulatory, or user-initiated deletion requirements. Existing work has largely focused on first-order optimization, where unlearning means that the post-deletion model is indistinguishable from the retrained ideal in terms of parameters. In this setting, the learner’s state is effectively summarized by its parameters, and deletion can be implemented with parameter perturbations [ 7 ] , partitioned retraining [ 1 ] , or approximate inverse updates with certificates that bound residual leakage [ 10 , 11 ] . Second-order optimizers make this difficult. They were designed to use curvature information to "quickly" gain information on the loss surface, but this additional state of information was never the focus of machine unlearning work. For algorithms like Online Newton Step (ONS), this internal geometry defines a time-varying metric on parameter space, encoding how past data inform future predictions. A deletion request targets not only the parameters but also the optimizer’s memory, raising the question of what it means to "forget" data when the learning dynamics themselves depend on a compressed record of the entire trajectory. This paper argues two things: that deleted information persists as volatility in the secon-order auxillary state, and that this volatility is reduced when the model undergoes geometric pertubation, mimicking the erasure of second-order information. Current formulations of certified machine unlearning are underspecified for stateful optimizers. Definitions that equate successful unlearning with parameter similarity ignore the possibility that higher-order states retain information about deleted observations, even when external performance appears unaffected. This complicates data privacy work that seeks to efficiently remove the data from a model while only measuring performance and parameter difference. We do not attempt to solve the problem of certified machine unlearning. We instead pose the problem as under-specified, and pose the challenge as a task for future research. 2 Related Work There is a substantial body of work on convex optimization, with applications in online learning, data deletion, and machine unlearning. We review the pieces most relevant to our work while acknowledging the breadth of research available. 2.1 Online Convex Optimization. Convex optimization methods aim to efficiently locate a global optimum of a convex objective, subject to user-specified constraints. In the online setting, observations are revealed incrementally to the learner, requiring the model to update its parameters sequentially rather than in batches. This is formalized in the online convex optimization problem, where performance is measured relative to some ideal comparator [ 2 ] . First-order methods such as gradient descent remain popular due to their simplicity and minimal memory requirements. Each update consists of observing a gradient, selecting a step size, and applying a recursive update rule. However, these methods discard nearly all historical information beyond the current iterate. Second-order methods, including Newton-style algorithms, trade additional memory and computation for faster convergence and improved conditioning. These methods explicitly retain curvature information, raising a fundamental design question: how much information from the optimization trajectory must be stored to ensure robust and efficient learning? 2.2 Machine Unlearning. The problem of data deletion was formalized rigorously in previous works. [ 6 , 11 , 7 ] , who propose proactive mechanisms that render learned models insensitive to the inclusion or exclusion of any single training point. Under this framework, no explicit deletion operation is required post hoc, as privacy is guaranteed by construction. Subsequent work explored practical implementations of proactive unlearning. Bourtoule et al. [ 1 ] propose partitioned training, where each model component is trained on a disjoint subset of the data and retrained upon deletion. While effective, these approaches incur substantial performance and retraining costs. Later work relaxes the requirement of perfect privacy in favor of approximate deletion guarantees. Ginart et al. frame unlearning as decremental learning, measuring success via distance in parameter space from an ideal model that never observed the deleted point [ 7 ] . Guo et al. propose a Newton-based deletion step that removes the influence of a data point in a single update [ 8 ] , with any residual leakage mitigated through calibrated noise [ 5 ] . These approximate guarantees significantly improve practicality but define unlearning narrowly as parameter matching to a counterfactual model. 2.3 Unlearning via Second-Order Approximations. Second-order methods are particularly attractive for unlearning due to their explicit modeling of curvature. Classical Newton methods require storing and inverting the Hessian, which scales quadratically with parameter dimension. Quasi-Newton methods such as BFGS alleviate this cost by maintaining low-rank approximations to the inverse Hessian. Nocedal and Wright formalized these techniques and demonstrated that curvature information can substitute for repeated gradient evaluations [ 3 ] . In the online setting, Mokhtari et al. proposed an online L-BFGS algorithm that retains a constant number of curvature pairs while achieving global convergence guarantees [ 9 ] . These methods emphasize efficiency and scalability but are typically analyzed under the assumption of monotonic data accumulation rather than adversarial deletions. 2.4 Online and Continuous Learning Recent work has emphasized the role of stateful optimizers , which reason over some compressed form of information, called a iterate. This iterate can be a preconditioner matrix, curvature estimates, or any sufficient information state (more on this in future work). These methods address limitations of memoryless algorithms, particularly in nonstationary environments, and have been shown to achieve improved convergence rates and stability in online learning settings. However, existing analyses of sequential learning largely assume that the optimization trajectory evolves monotonically as new data arrive. Deletions, when considered at all, are typically treated as retraining events or as perturbations to the model parameters. This perspective neglects the fact that stateful optimizers accumulate higher-order information that is itself shaped by the full training history. While much prior work evaluates unlearning by comparing parameter vectors to a counterfactual model, this criterion is insufficient for stateful learners whose internal geometry encodes additional information. Sequential deletions can induce persistent discrepancies between the realized optimizer state and its counterfactual counterpart, even when parameter estimates appear similar. Our work departs from parameter-centric analyses by explicitly studying the evolution, distortion, and hysteresis of optimizer state under online deletions. 3 Problem Formulation We study online convex optimization under adversarial deletions.