Recipes for Pre-training LLMs with MXFP8

Asit Mishra, Dusan Stosic, Simon Layton · 2025

Read Recipes for Pre-training LLMs with MXFP8 by Asit Mishra, Dusan Stosic, Simon Layton (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08027.txt.

Description

Asit Mishra, Dusan Stosic, and Simon Layton explain in 2025 how to pre-train large language models using a compact number format called MXFP8. Using fewer bits for model weights and related data has become a standard way to get more work out of modern graphics chips without losing accuracy.

The authors review the choices that make the format work well in practice. They focus on the MXFP8-E4M3 data type and one number conversion method, and show that this pairing trains models cleanly. Their recipes give training teams a tested set of settings for the newest NVIDIA Blackwell chips.

Catalog record

CREATORAsit Mishra, Dusan Stosic, Simon Layton
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-08027

Browse the full library · About the Tower of Babel