Unpacking Softmax: How Temperature Drives Representation Collapse, Compression and Generalization
Wojciech Masarczyk, Mateusz Ostaszewski, Tin Sum Cheng, Tomasz Trzciński, Aurelien Lucchi, Razvan Pascanu · 2025
Read Unpacking Softmax: How Temperature Drives Representation Collapse, Compression and Generalization by Wojciech Masarczyk, Mateusz Ostaszewski, Tin Sum Cheng, Tomasz Trzciński, Aurelien Lucchi, Razvan Pascanu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01562.txt.
Introduction
Wojciech Masarczyk and co-authors studied in 2025 how the temperature of the softmax function shapes what neural networks learn. Softmax turns raw scores into probabilities and sits inside both classifiers and attention layers. Its wider effect on learned features, though, has been poorly understood.
The authors name a pattern they call rank deficit bias: networks with softmax tend to find solutions whose rank is far below the number of classes. That bias depends on the size of the raw scores, which temperature directly controls. The paper shows how to use that effect rather than suffer it, linking collapse of features, compression, and generalization. Readers get a new dial to reason about: turn the temperature, and you change how compactly a model represents its classes.
Catalog record
| CREATOR | Wojciech Masarczyk, Mateusz Ostaszewski, Tin Sum Cheng, Tomasz Trzciński, Aurelien Lucchi, Razvan Pascanu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unpacking Softmax: How Temperature Drives Representation Collapse, Compression and Generalization Wojciech Masarczyk, Mateusz Ostaszewski, Tin Sum Cheng, Tomasz Trzciński, Aurelien Lucchi, Razvan Pascanu Abstract The 𝚜𝚘𝚏𝚝𝚖𝚊𝚡 \mathtt{softmax} function is a fundamental building block of deep neural networks, commonly used to define output distributions in classification tasks or attention weights in transformer architectures. Despite its widespread use and proven effectiveness, its influence on learning dynamics and learned representations remains poorly understood, limiting our ability to optimize model behavior. In this paper, we study the pivotal role of the 𝚜𝚘𝚏𝚝𝚖𝚊𝚡 \mathtt{softmax} function in shaping the model’s representation. We introduce the concept of rank deficit bias — a phenomenon in which softmax-based deep networks find solutions of rank much lower than the number of classes. This bias depends on the 𝚜𝚘𝚏𝚝𝚖𝚊𝚡 \mathtt{softmax} function’s logits norm, which is implicitly influenced by hyperparameters or directly modified by 𝚜𝚘𝚏𝚝𝚖𝚊𝚡 \mathtt{softmax} temperature. Furthermore, we demonstrate how to exploit the 𝚜𝚘𝚏𝚝𝚖𝚊𝚡 \mathtt{softmax} dynamics to learn compressed representations or to enhance their performance on out-of-distribution data. We validate our findings across diverse architectures and real-world datasets, highlighting the broad applicability of temperature tuning in improving model performance. Our work provides new insights into the mechanisms of 𝚜𝚘𝚏𝚝𝚖𝚊𝚡 \mathtt{softmax} , enabling better control over representation learning in deep neural networks. 1 1 footnotetext: Warsaw University of Technology, Poland 2 2 footnotetext: University of Basel, Switzerland 3 3 footnotetext: Mila, Québec AI Institute, Canada 4 4 footnotetext: IDEAS Research Institute, Poland 5 5 footnotetext: Tooploox, Poland 1 1 footnotetext: Corresponding author: [email protected] 1 Introduction The \mathtt{softmax} function defined in Equation 1 , with a hyperparameter T>0 as the temperature , is a cornerstone of deep learning and is primarily used in tasks such as classification and text generation to transform raw model outputs into probability distributions with maximum entropy. By amplifying the largest values and diminishing smaller ones, \mathtt{softmax} enables models to make confident predictions, making it essential for decision-making among multiple options. However, its "winner-takes-all" nature liu1999winner ; elfadel1993softmax ; Peterson89 can sometimes lead to training challenges darcet2023vision ; hoffmann2023softmax ; shen2023softmax ; zhai2023softmax . \operatorname{softmax}_{T}(\mathbf{e})=\left[\begin{array}[]{llc}\frac{\exp\left(e_{1}/T\right)}{\sum_{k}\exp\left(e_{k}/T\right)}&\cdots&\frac{\exp\left(e_{n}/T\right)}{\sum_{k}\exp\left(e_{k}/T\right)}\end{array}\right] (1) Originally, the \mathtt{softmax} function was used primarily as the final layer in classification tasks to produce normalized class probabilities. However, with the introduction of the self-attention mechanism in transformer models vaswani2017attention , \mathtt{softmax} became central to internal computations—particularly in attention blocks used for text generation, and later, for image classification dosovitskiy2020image . Despite various alternatives proposed for the attention mechanism saratchandran2024rethinking ; ramapuram2024theory , \mathtt{softmax} -based attention remains the dominant approach in modern architectures. This enduring popularity has spurred extensive research into the implicit trade-offs of using the \mathtt{softmax} activation, such as its impact on the sharpness of output distributions velickovic2024softmax and its role in normalizing activations and gradients saratchandran2024rethinking . Yet, the deeper effects of \mathtt{softmax} on model behavior—particularly its influence on learned representations and generalization—remain poorly understood. This gap motivates our central research question: How does \mathtt{softmax} shape neural network representations? Our answer to this question unfolds into four main contributions of our work, visualized in Figure 1 : 1. We introduce rank-deficit bias —a novel phenomenon where the \mathtt{softmax} biases learning to converge to solutions whose representation has rank much lower than the number of classes. This departs from the behavior predicted by Neural Collapse papyan2020neuralcollapse , a framework that describes the emergence of highly structured, full-rank representations at convergence. 2. We show that the logit’s norm at initialization drives the difference in network behavior, influencing representations’ compactness, out-of-distribution generalization, and out-of-distribution detection. 3. We illustrate that the \mathtt{softmax} temperature directly changes logits norm and gives direct control over the training trajectory, which can be used to increase the model’s OOD performance or compress the model’s size. 4. Beside the \mathtt{softmax} temperature, we identify other hyperparameters that play a similar role. This observation gives practitioners more flexibility to control the outcome of the learning process. 2 Unpacking Softmax Figure 1 : Logits norm shapes the representations of neural networks leading to the best performance achieved at earlier layers for in distribution data when trained with low logits norm (right) compared to the baseline (left). These condensed representations trigger representation collapse , which harms generalization on out-of-distribution data . The plot presents the accuracy of linear probes attached to ResNet-34 trained on CIFAR-100 with different \mathtt{softmax} temperatures, the OOD dataset is SVHN. Analogous plots with different architectures and datasets can be found in the Appendix C . The \mathtt{softmax} function is a ubiquitous component of modern neural networks, yet its role in shaping learned representations—particularly through its temperature parameter—remains underexplored. In this section, we investigate how variations in the logits norm at initialization, induced by the \mathtt{softmax} temperature, systematically affect representation learning and generalization. We present empirical evidence showing that lower logits norms lead to more compressed internal representations, reflected in reduced effective depth and rank, and, surprisingly, worse generalization to out-of-distribution (OOD) data. To quantify these effects, we introduce a suite of diagnostic metrics and evaluate them across diverse architectures and datasets. These observations lay the empirical foundation for our theoretical analysis in Section 3 . 2.1 Empirical evidence To thoroughly examine the phenomenon, we designed a setup to include the most commonly used architectures and datasets, and designed several metrics to quantify its strength. Architectures and Datasets We use 4 different architecture groups: MLP, VGG, ResNet, and VIT, trained on a diversified set of datasets: CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet-1k, to assess how broadly the phenomenon applies. The details about architectures, datasets, and hyperparameters can be found in the Appendix A . To quantitatively measure the effect, we define the following estimates: effective depth, OOD generalization loss, and solutions rank based on the linear probing accuracy and numerical rank. Effective depth ( \kappa ): measures the ratio between the first layer of the model’s that achieves at least 99% of the last layer’s model accuracy using linear probing and the total number of layers in the model. The smaller the value, the more efficient the model. OOD generalization loss ( \rho ): is defined as the normalized difference between the best OOD accuracy (across the layers) and the OOD accuracy at the final layer using linear probes. The value is non-negative, and the smaller the value, the better. Solutions rank (SR): We compute the numerical rank of the pre-softmax representations matrix (logits matrix). Using the spectrum of the matrix, we estimate the numerical rank of the given representation matrix as the number of singular values above a threshold \gamma . * * * We use the default value of this hyperparameter proposed in PyTorch paszke2019pytorch . We begin our analysis by examining how varying the logits norm at initialization affects training dynamics and representation quality. Table 1 compares baseline architectures with their low-logit-norm counterparts, revealing clear shifts in representation structure across different datasets and model families. Even when final classification accuracies are comparable, models trained with lower logits norm exhibit systematically lower solution rank, reduced effective depth ( \kappa ), and increased OOD generalization loss ( \rho ). These patterns highlight a trade-off: lower logits norm promotes more compressed internal representations—suggesting more efficient use of model capacity—but also leads to performance degradation on OOD data, especially in deeper layers.