Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis

Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley · 2025

Read Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis by Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11581.txt.

Introduction

Akarsh Kumar, Jeff Clune, Joel Lehman, and Kenneth Stanley question a cheery belief about deep learning: that better scores mean better inner order. They test it on a tiny task, making one image, where each neuron's work can be drawn and seen. Their 2025 paper compares networks trained the usual way with networks bred by an open-ended search.

Both nets made the same image, but their innards split apart. The usual nets showed a fractured, tangled code, where single ideas smear across neurons in messy ways. The bred nets kept a cleaner, factored code. The authors warn the tangle may sap big models' power to generalize, create, and keep learning. They call for work to name and tame it.

Catalog record

CREATORAkarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11581

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley Abstract Much of the excitement in modern AI is driven by the observation that scaling up existing systems leads to better performance. But does better performance necessarily imply better internal representations? While the representational optimist assumes it must, this position paper challenges that view. We compare neural networks evolved through an open-ended search process to networks trained via conventional stochastic gradient descent (SGD) on the simple task of generating a single image. This minimal setup offers a unique advantage: each hidden neuron’s full functional behavior can be easily visualized as an image, thus revealing how the network’s output behavior is internally constructed neuron by neuron. The result is striking: while both networks produce the same output behavior, their internal representations differ dramatically. The SGD-trained networks exhibit a form of disorganization that we term fractured entangled representation (FER). Interestingly, the evolved networks largely lack FER, even approaching a unified factored representation (UFR). In large models, FER may be degrading core model capacities like generalization, creativity, and (continual) learning. Therefore, understanding and mitigating FER could be critical to the future of representation learning. Code and additional data: https://github.com/akarshkumar0101/fer Figure 1 : Overview. An open-ended search process (left) from the Picbreeder experiment yielded a neural network that outputs the image of a skull. Conventional SGD (right) is able to learn to produce precisely the same image, but by following a completely different path through function space during learning. However, peering inside these networks with identical output behavior reveals a radically different style of representation, raising questions about the adaptability of networks (including large models) trained through conventional SGD, which can impact generalization, creativity, and (continual) learning. 1 Introduction A neural network’s knowledge, generalization, creativity, continual learning ability, and overall potential are ultimately determined by its internal representations. These representations—how it models the world—are encoded through the structure of its neural circuitry (connection weights). Even as researchers increasingly probe the internal representations of neural networks (NNs) ( Yosinski et al., 2015 ; Nguyen et al., 2016 ; Olah et al., 2017 ; Elhage et al., 2022 ; Olsson et al., 2022 ; Lindsey et al., 2025 ) , an implicit philosophy of representational optimism has emerged—rarely stated outright but carrying profound implications. The hope of the representational optimist is that, as data and compute are scaled, good representations naturally develop on their own. While scaling clearly improves the performance of the representations on downstream tasks ( Kaplan et al., 2020 ; Bubeck et al., 2023 ) , less attention has been given to understanding the nature of these representations as models grow. Of course, even representational optimists still want to understand internal representations (e.g. through mechanistic interpretability as in Sharkey et al., 2025 ), but the motivation is more through a desire to understand how the networks work ( Olah et al., 2017 ; Yosinski et al., 2015 ) , how they can be controlled, and their level of alignment ( Bereska and Gavves, 2024 ; Lindsey et al., 2025 ) than through any concern that their representations may be fundamentally flawed. At the heart of representational optimism is the belief that representation improves with scaling in deep learning ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ; Huh et al., 2024 ) . For example, the performance leaps from GPT-2 ( Radford et al., 2019 ) to GPT-3 ( Brown et al., 2020 ) and GPT-4 ( Achiam et al., 2023 ) suggest that the underlying representation should also be improving. As Brown et al., 2020 state in the introduction to the technical report introducing GPT-3, where they place focus on the implications for representation (boldface added for emphasis): Recent years have featured a trend towards pre-trained language representations in NLP systems, applied in increasingly flexible and task-agnostic ways for downstream transfer. First, single-layer representations were learned using word vectors and fed to task-specific architectures, then RNNs with multiple layers of representations and contextual state were used to form stronger representations (though still applied to task-specific architectures), and more recently pre-trained recurrent or transformer language models… The implication here that representation is improving makes intuitive sense; after all, how could it be possible that performance improves without the underlying representation also getting better ( Bengio et al., 2013 ) ? Because of that, the thinking goes, as models are scaled further, they will continue improving, and little worry need arise, at least with respect to representation. Moreover, the mechanistic interpretability literature often studies a single neuron ( Yosinski et al., 2015 ; Nguyen et al., 2016 ; Carter et al., 2019 ) or a small group of neurons, reinforcing representational optimism by inadvertently diverting attention away from potential problems at a more holistic rather than local level ( Templeton et al., 2024 ) . Even the representation learning literature mostly focuses on only the final layer features ( Chen et al., 2020 ; Grill et al., 2020 ; He et al., 2020 ; Radford et al., 2021 ; Caron et al., 2021 ) . Yet what if something important is actually missing in the learned representation? The main contribution of this position paper is to raise the possibility that in practice representation might not simply work itself out well by scaling after all (Figure 1 ). To address the question of whether representations may be flawed even when benchmark performance is good , we shift the focus from studying individual neurons to studying the broader, holistic representational strategy across the entire network. By doing so, we are able to make a distinction between an adequate representation that solves a task and an ideal representation of that solution. The core of this concern is crystallized through a novel hypothesis that is based on a set of little-known observations of neural network representation that arose initially in the fields of neuroevolution and artificial life ( Lehman and Stanley, 2011 ; Lehman and Stanley, 2010 ; Secretan et al., 2011 ; Clune et al., 2011 ; Woolley and Stanley, 2011 ; Huizinga et al., 2018 ; Lehman et al., 2025 ) : The Fractured Entangled Representation (FER) hypothesis. We hypothesize that when deep learning models are trained to achieve a specific objective, typically through backpropagation and stochastic gradient descent, the resultant representation tends to be fractured, which means that information underlying the same unitary concepts (e.g. how to add numbers) is split into disconnected pieces. Importantly, these pieces then become redundant as a result of their fracture: they are separately invoked in different contexts to model the same underlying concept, when ideally, a single instance of the concept would have sufficed. In other words, where there would ideally be the reuse of one deep understanding of a concept, instead there are different mechanisms for achieving the same function. At the same time, these fractured (and hence redundant) functions tend to become entangled with other fractured functions, which means that behaviors that should be independent and modular end up influencing each other in idiosyncratic and inappropriate ways. For example, a set of neurons within an image generator that change hair color might also cause the

Read the complete text