Neural Networks Learn Distance Metrics

Alan Oursland · 2025

Read Neural Networks Learn Distance Metrics by Alan Oursland (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02103.txt.

Introduction

Alan Oursland tested a hypothesis about how neural networks represent the world in 2025: by distance rather than by brightness. In a distance-based representation, a neuron's small activation means the input sits close to a learned prototype, while intensity-based schemes read meaning off raw activation size. Which style a network naturally prefers could shape how architectures ought to be designed, yet the question is rarely isolated.

Oursland trained six variants of digit-recognition networks on the MNIST dataset, each constrained to learn in one style or the other, and found the underlying representation measurably changed performance. He then developed a geometric framework explaining why, and introduced OffsetL2, an architecture built on Mahalanobis distance formulas, which behaved as the framework predicted. The work argues distance-based learning deserves a central place in network design.

Catalog record

CREATORAlan Oursland
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02103

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural Networks Learn Distance Metrics Alan Oursland Abstract Neural networks may naturally favor distance-based representations, where smaller activations indicate closer proximity to learned prototypes. This contrasts with intensity-based approaches, which rely on activation magnitudes. To test this hypothesis, we conducted experiments with six MNIST architectural variants constrained to learn either distance or intensity representations. Our results reveal that the underlying representation affects model performance. We develop a novel geometric framework that explains these findings and introduce OffsetL2, a new architecture based on Mahalanobis distance equations, to further validate this framework. This work highlights the importance of considering distance-based learning in neural network design. 1 Introduction Neural networks have transformed machine learning through their ability to learn complex, hierarchical representations, traditionally interpreted through intensity-based representations where larger activation values signify stronger feature presence. This interpretation, originating from the McCulloch-Pitts neuron McCulloch and Pitts (1943) and Rosenblatt’s perceptron Rosenblatt (1958) , underlies modern deep learning LeCun et al. (2015) . However, our theoretical understanding of neural networks’ internal mechanisms remains limited Lipton (2018) ; Montavon et al. (2018) , particularly regarding how they represent and process features. In our prior work Oursland (2024a) , we argued that networks may naturally learn distance-based representations, where smaller activations indicate proximity to learned prototypes. This reinterpretation challenges the traditional intensity-based paradigm and provides a statistical foundation rooted in principles like the Mahalanobis distance Mahalanobis (1936) ; McLachlan (2019) . Empirical evidence supports this theory. In our work Oursland (2024b) , we demonstrated that distance-based metrics play a crucial role in how networks learn and utilize features, suggesting the need to rethink the fundamental nature of representations. Core Questions. This paper investigates: (1) whether neural networks naturally prefer distance-based or intensity-based representations, (2) how architectural choices shape these representational biases, and (3) what geometric and statistical principles underlie these preferences. Contributions. We examine neural network behavior through distance and intensity representations via: (1) A theoretical framework formalizing the distinction between these representations, (2) empirical analysis of six architectural variants, revealing mechanisms behind dead node creation and geometric performance limitations, and (3) introduction of OffsetL2, a novel architecture validating our framework through strong, stable performance. The remainder of this paper reviews related work (Section 2 ), establishes theoretical foundations (Section 3 ), presents our experimental design (Section 4 ) and findings (Section 5 ), and discusses implications (Section 6 ). 2 Related Work Our work builds upon recent advances in understanding neural networks through statistical distance metrics. Our prior work Oursland (2024a) demonstrated how linear layers with Absolute Value (Abs) activations approximate the Mahalanobis distance Mahalanobis (1936) . We established a mathematical foundation for distance-based representations, showing that such layers naturally encode distance relationships rather than intensity-based activations. Empirical studies further reinforce this perspective. In Oursland (2024b) , we demonstrated that networks with ReLU and Abs activations exhibit particular sensitivity to perturbations affecting distance relationships in the feature space, suggesting that distance metrics play a fundamental role in how networks process information. Alternative approaches to incorporating distance metrics in neural networks include Radial Basis Function (RBF) networks Broomhead and Lowe (1988) ; Park and Sandberg (1991) , which use distances from learned centers for classification, Siamese networks Bromley et al. (1994) ; Hadsell et al. (2006) , which learn embeddings where distances represent similarity, and Learning Vector Quantization (LVQ) Kohonen (1995) , which explicitly models class prototypes and uses distance-based classification. Contrastive learning Chen et al. (2020) ; He et al. (2020) emphasizes the importance of learning representations where distances reflect semantic similarity, although those methods typically rely on carefully constructed training objectives rather than inherent architectural biases. While specialized architectures like RBF and LVQ demonstrate the effectiveness of explicitly encoding distances, they remain largely confined to specific applications rather than general-purpose deep learning. Complementing these distance-centric views, geometric interpretations of neural computation offer valuable insights for understanding internal representations Montavon et al. (2018) ; Olah et al. (2017) . These approaches analyze hyperplanes and decision boundaries to explain how networks partition and represent data Lipton (2018) ; Erhan et al. (2009) , though they typically focus on networks trained under standard intensity-based assumptions. This work bridges the gap between distance-based and geometric interpretations by investigating how architectural choices influence the emergence of distance-based representations. 3 Background This section introduces the theoretical framework for understanding how neural networks represent data through either distance-based or intensity-based methods, providing context for our experimental analysis. 3.1 Features, Representations, and Distance Metrics Neural networks learn features through their internal representations. For the purposes of this paper, we define features as inherent properties of the data that can be quantified statistically, and representations are the compositions of these features learned by a node to generate its output. Traditional intensity-based representations encode features through activation magnitudes, while distance-based representations encode features through proximity to learned prototypes. In our discussion, we primarily consider prototypes as the features of interest, as they provide a structured way to quantify similarity and influence learned representations. We argue that neural networks fundamentally learn distance features that quantify similarity between data points and learned prototypes. What appear to be intensity-based representations can be reinterpreted as disjunctive sets of distance features, corresponding to Disjunctive Normal Form (DNF) in Boolean algebra Post (1921) . For example, an intensity representation for class a implicitly learns \lnot(b\lor c)=\lnot b\land\lnot c , where b and c represent distances to other classes. This logical interpretation has connections to work exploring the relationship between neural networks and Boolean formulas Anthony (2003) . In contrast, distance-based representations directly encode proximity to prototypes as conjunctive sets (Conjunctive Normal Form). For class a , this simply requires a small distance to prototype a , represented logically as a . This more directly captures the underlying statistical relationships in the data. 3.2 Distance vs. Intensity Representations Intensity-based interpretations, while lacking precise mathematical definition, are deeply embedded in the foundations of deep learning Goodfellow et al. (2016) . These interpretations treat larger activations as indicating stronger feature presence, fundamentally shaping how we train and interpret networks. The use of one-hot encoded labels for classification directly encodes this assumption - the correct class should produce the highest activation while all others should be suppressed. This intensity-based paradigm appears throughout modern practice: cross-entropy loss encourages larger activations for correct classes, feature visualization Erhan et al. (2009) ; Olah et al. (2017) assumes maximum activations represent learned features, and even basic image processing interprets pixel intensities as direct measures of feature strength. However, this pervasive intensity-based interpretation remains an assumption imposed on neural networks rather than an inherent property. Distance-based representations offer an alternative framework, interpreting smaller activations as indicating similarity to learned prototypes. This aligns with statistical metrics like the Mahalanobis distance Mahalanobis (1936) , where the activation f(x) is inversely proportional to the distance between input x and prototypes \mu : f(x)=|W(x-\mu)|_{p}, (1) where W is a learned scaling matrix and p denotes the norm. Common activation functions naturally support this view: the absolute value function directly represents distance, while ReLU implicitly encodes distance through the relationship Abs(x)=ReLU(x)+ReLU(-x) . This framework emphasizes geometric relationships in the latent space rather than activation magnitudes. This perspective aligns with research on visualizing and understanding the

Read the complete text