Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis

Damien Martins Gomes · 2025

Read Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis by Damien Martins Gomes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20096.txt.

Introduction

Damien Martins Gomes presents a thesis on faster deep learning training in a 2025 paper. Simple first order methods such as Adam rule training today, but they use only a little curve data. Second order methods can converge better, yet each step costs far more. That cost has kept them from wide use.

The thesis offers AdaFisher, a second order optimizer built on a block form of the Fisher information matrix. It aims to keep the gains of second order methods while staying practical. Tests on image and language tasks beat leading optimizers in accuracy and speed, the author reports. The tool is also stable when settings change.

Catalog record

CREATORDamien Martins Gomes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE229 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20096

Browse the full library · About the Tower of Babel