Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis
Damien Martins Gomes · 2025
Read Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis by Damien Martins Gomes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20096.txt.
Introduction
Damien Martins Gomes presents a thesis on faster deep learning training in a 2025 paper. Simple first order methods such as Adam rule training today, but they use only a little curve data. Second order methods can converge better, yet each step costs far more. That cost has kept them from wide use.
The thesis offers AdaFisher, a second order optimizer built on a block form of the Fisher information matrix. It aims to keep the gains of second order methods while staying practical. Tests on image and language tasks beat leading optimizers in accuracy and speed, the author reports. The tool is also stable when settings change.
Catalog record
| CREATOR | Damien Martins Gomes |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 229 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20096