Representation biases: will we achieve complete understanding by analyzing representations?
Andrew Kyle Lampinen, Stephanie C. Y. Chan, Yuxuan Li, Katherine Hermann · 2025
Read Representation biases: will we achieve complete understanding by analyzing representations? by Andrew Kyle Lampinen, Stephanie C. Y. Chan, Yuxuan Li, Katherine Hermann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22216.txt.
Introduction
Andrew Lampinen, Stephanie Chan, and Yuxuan Li lead a team that warns that learned representations can skew what analysts think a brain or a model is doing. Their piece came out in two thousand twenty-five. Comparing inner patterns is a common route to grasp how a system works. New results in machine learning show these patterns favor simple features.
Linear features show up strongly and steadily, while complex ones show up weakly and in patches. Common tools then pick up that skew. The authors show the skew in tools such as regression and similarity analysis. A homomorphic encryption case shows pattern and computation can come apart in full. Brains may hold the same traps when they are compared with models. The piece calls for care in such claims. To grasp a system, we need more than a read of its features.
Catalog record
| CREATOR | Andrew Kyle Lampinen, Stephanie C. Y. Chan, Yuxuan Li, Katherine Hermann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22216
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Representation biases: will we achieve complete understanding by analyzing representations? Andrew Kyle Lampinen, Stephanie C. Y. Chan, Yuxuan Li, Katherine Hermann Abstract A common approach in neuroscience is to study neural representations as a means to understand a system—increasingly, by relating the neural representations to the internal representations learned by computational models. However, a recent work in machine learning ( Lampinen et al. 2024 ) shows that learned feature representations may be biased to over-represent certain features, and represent others more weakly and less-consistently. For example, simple (linear) features may be more strongly and more consistently represented than complex (highly nonlinear) features. These biases could pose challenges for achieving full understanding of a system through representational analysis. In this perspective, we illustrate these challenges—showing how feature representation biases can lead to strongly biased inferences from common analyses like PCA, regression, and RSA. We also present homomorphic encryption as a simple case study of the potential for strong dissociation between patterns of representation and computation. We discuss the implications of these results for representational comparisons between systems, and for neuroscience more generally. \reportnumber keywords Representation, representation analysis, computation, computational neuroscience, deep learning, statistical methods Figure 1 : What do representations tell us about a system? Here, we discuss some experiments by Lampinen et al. 2024 that illustrate challenges in making inferences about a system from its representations. The experiments involve training models on controlled datasets mapping from many input features to outputs involving simpler or more complex functions of the input features. The representations that models learn from these datasets show substantial biases towards certain outputs over others (e.g. towards simple linear functions of the input over complex nonlinear ones), even when the model has reliably learned both. These biases raise questions about a linking assumption that underlies many analytic methods and common analyses in neuroscience: that variance explained in the neural representations (or activity) is a good proxy for the importance of a signal. 1 Introduction A central approach of neuroscience is analyzing patterns of neural representation to learn about a system ( Kriegeskorte and Diedrichsen 2019 ) . In particular, computational neuroscience has increasingly relied on relating patterns of neural activity to the internal representational structures of computational models ( Churchland and Sejnowski 1990 ; Kriegeskorte et al. 2008 ; Sucholutsky et al. 2023 ; Feather et al. 2025 ) . However, there are philosophical questions about how to justify interpreting internal activity as representations ( Shea 2018 ; Fallon et al. 2023 ; Cao 2022b ) and conceptual and practical challenges to understanding a system through analyzing its internal activity or representations ( Poldrack 2006 ; Marom et al. 2009 ; Jonas and Kording 2017 ; Ritchie et al. 2019 ; Sexton and Love 2022 ; Dujmovic et al. 2024 ) . Here, we use case studies from recent work in machine learning to illustrate some practical challenges to understanding a system’s function by studying its internal representations (see Fig. 1 for an overview)—and discuss the implications for neuroscience. In particular, we focus on the results of Lampinen et al. 2024 . In this work, the authors study the relationship between patterns of representation and computation in machine learning models, using controlled experiments. The authors identify substantial biases in the learned representations: some features are much more strongly represented than others, even if they play similar computational roles in the system’s behavior. These representation biases mean that common analytic methods—such as Principal Component Analysis (PCA), Representational Similarity Analysis ( Kriegeskorte et al. 2008 , RSA;) , and linear regression—may be biased towards capturing some computational features over others. Thus, the many types of neuroscience experiments that use these methods like these to study a system’s representations or activity may provide a biased picture of its computations. We first briefly review the prior literature on representations and computations in cognitive science and neuroscience. We then turn to our central examples illustrating representation biases, and how they impact commmon representational analyses. We then briefly present a case study of homomorphic encryption, which illustrates how strong the dissociation between representations and the system’s function can be. We close by discussing the implications of our results, and the questions they raise. 1.1 Background In a foundational paper, Marr 1982 famously posited three levels of analysis for understanding an information processing system—computational, algorithmic, and implementational. In the years since, there has been substantial discussion about the (non)independence of the levels, their interactions and joint constraints, and what they imply about representation ( McClamrock 1991 ; Churchland and Sejnowski 1990 ; O’Brien and Opie 2009 ; Stevens 2012 ; Piccinini and Shagrir 2014 ; Hardcastle and Hardcastle 2015 , e.g.) . In the philosophical literature, similarly, the question of the relationship between representation and computation has been somewhat fraught ( Eliasmith and Anderson 2003 ; Piccinini 2008 ; Shea 2018 , e.g.) . There have also been arguments from a variety of perspectives that intelligence is best understood (or engineered) by focusing on the interaction between a system and its environment ( Clark 1998 ; Cisek 1999 ; Brooks 1991 ; Varela et al. 2017 ) . These approaches therefore tend to think of “representations” as more dynamic and entangled with the environment state rather than as static ways of encoding information. In addition to the conceptual debates about representation, there has been substantial debate about the empirical relationship between the representations of a system and its computational mechanisms or behavior. Several works have highlighted particular case studies where applying neuroscience analyses to well-understood computational systems does not yield the expected results ( Marom et al. 2009 ; Jonas and Kording 2017 ) . In machine learning, aside from the main paper we discuss, a number of other works have demonstrated that there can be substantial dissociations between the internal representational structure of a system and its computations or behavior ( Hermann and Lampinen 2020 ; Braun et al. 2025 ) —especially under distribution shift ( Friedman et al. 2023 ; Dujmovic et al. 2024 ) . There has also been a longer-standing discussion about the more concrete issue of the relationship between neural activity and cognitive mechanisms. For example, many researchers have noted that making inferences about cognitive processes or information content from downstream effects like fMRI activity ( Poldrack 2006 ) or decoding ( Ritchie et al. 2019 ) can be logically invalid. Others have noted the possibility that representational correlations do not necessarily imply computational correspondence, and therefore emphasized more stringent measures ( Sexton and Love 2022 ) . As in the conceptual work above, some neuroscientists have taken more provocative stances that “representation” is generally the wrong way to understand neural activity ( Freeman III and Skarda 1990 , e.g.) . However, there is clearly still information to be gained from representational analyses, and thus many researchers take a pragmatic perspective on representation in neuroscience: interpreting activity measures as a representation insofar as it satisfies conditions of reliable decodability together with causal efficacy ( Cao 2022b ; cf. Poldrack 2021 ; Baker et al. 2022 ). That is, a pattern (or distribution) of activity can be considered a representation insofar as we can consistently decode the expected information content from