Contextures: Representations from Contexts

Runtian Zhai, Kai Yang, Che-Ping Tsai, Burak Varıcı, Zico Kolter, Pradeep Ravikumar · 2025

Read Contextures: Representations from Contexts by Runtian Zhai, Kai Yang, Che-Ping Tsai, Burak Varıcı, Zico Kolter, Pradeep Ravikumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01557.txt.

Introduction

Runtian Zhai and co-authors present a machine learning theory paper, published in 2025, asking what foundation models actually learn inside. Their answer is a theory they call contexture: many learning methods, the authors prove, boil down to capturing how each input associates with a context around it. In that view, a learned representation approximates the top patterns of an operator built from those associations.

The single lens covers supervised, self supervised, and manifold learning alike. Representations that capture the contexture are shown to be optimal for any task that fits the same context. One sharp implication is that past a certain size, bigger models gain little; better contexts, not more parameters, are the lever. The paper also proposes a score for a context's usefulness.

Catalog record

CREATORRuntian Zhai, Kai Yang, Che-Ping Tsai, Burak Varıcı, Zico Kolter, Pradeep Ravikumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE124 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contextures: Representations from Contexts Runtian Zhai, Kai Yang, Che-Ping Tsai, Burak Varıcı, Zico Kolter, Pradeep Ravikumar Abstract Despite the empirical success of foundation models, we do not have a systematic characterization of the representations that these models learn. In this paper, we establish the contexture theory. It shows that a large class of representation learning methods can be characterized as learning from the association between the input and a context variable . Specifically, we show that many popular methods aim to approximate the top- d d singular functions of the expectation operator induced by the context, in which case we say that the representation learns the contexture . We demonstrate the generality of the contexture theory by proving that representation learning within various learning paradigms—supervised, self-supervised, and manifold learning—can all be studied from such a perspective. We also prove that the representations that learn the contexture are optimal on those tasks that are compatible with the context. One important implication of the contexture theory is that once the model is large enough to approximate the top singular functions, further scaling up the model size yields diminishing returns. Therefore, scaling is not all we need, and further improvement requires better contexts. To this end, we study how to evaluate the usefulness of a context without knowing the downstream tasks. We propose a metric and show by experiments that it correlates well with the actual performance of the encoder on many real datasets. 1 Introduction Representation learning underpins the modern deep learning revolution, leading up to the remarkable recent successes of foundation models [ BHA+21 ] . But a critical question that has remained unanswered to a satisfactory extent is: why are these models learning anything useful, or perhaps even what representations are these models learning? Unlike classical statistical learning theory, where there is no mystery regarding the statistical estimand, the very target itself is unclear in representation learning. For example, what are the representations that BERT [ DCLT19 ] —trained to do cloze tests—is learning, and why are they useful in understanding the sentiment of user reviews on Netflix? What representations do deep neural networks learn, and why are they useful if they cause neural collapse [ PHD20 ] , where deep representations could collapse to a few clusters? Do the many different self-supervised learning methods [ BIS+23 ] all learn similar or disparate representations? The responses to these questions are often muddled. Many analyses are conflated with the mystery of deep learning generalization—the ability of large neural networks to learn function approximations that generalize to unseen points. However, this is a very different problem from the mechanism of representation learning. Our focus is on what representation learning (also known as “pretraining” in the context of foundation models) aims to capture, and why it can be applied to tasks completely different from the objectives used to train the representations. Another way this question is muddled is by recourse to scaling. A popular viewpoint argues that even if the encoder performs poorly on one task, increasing the model size while keeping everything else the same could allow better performance to “emerge” [ WTB+22 ] . However, substantial evidence suggests that certain abilities cannot emerge solely from scaling. Additional training signals, such as alignment [ OWJ+22 ] , are necessary. The above questions are naturally interesting to learning theorists, but why should the broader machine learning community care about understanding the mechanism of representation learning, if empirical success seems to be always achievable with existing approaches by scaling up the model size, an empirical observation known as scaling laws [ KMH+20 ] ? This is because sustainable success or progress is not always guaranteed. Ilya Sutskever recently remarked that “pretraining as we know it will end” [ Sut24 ] , largely because the current pretraining paradigm is producing diminishing returns. Understanding what representations are learned by foundation models is crucial for designing future generations of pretraining methods, and this is how this field can make scientific progress. In this work, we establish the contexture theory , which provides a unified lens for inspecting a large class of representation learning methods. The central argument of the contexture theory is that representations are learned from the association between the input X and a context variable A . This framework is general enough to encompass a wide variety of learning paradigms, as we demonstrate in Section 3 . Now, suppose we are given a context variable A along with X , how should we learn the representation? In Section 4 we prove that the optimal method is to approximate the top singular functions of the expectation operator induced by the context, in which case we say that the encoder “ learns the contexture ”. Such an encoder is optimal as long as the task is known to be compatible with the context, and in Section 4 we define a quantitative measurement of such compatibility. Our theory implies that the main consequence of making the model large is that the learned representation space will be brought closer to the space spanned by the top- d singular functions of the expectation operator, which we empirically verify in Section 4.2 . Once the two spaces are close enough, further scaling will yield diminishing returns. We envision that future breakthroughs in pretraining require context scaling , where better contexts are learned from data, instead of heuristics. Taking a first step towards context scaling, in Section 5 we study how to evaluate contexts. This is a prerequisite because if we cannot even determine which contexts are good, then there will be no way to create better contexts. The key takeaway from our analysis is that for a context to be useful (meaning that it can lead to good representations), the association between X and A should be moderate—neither too strong nor too weak. For example, if A=X , then their association is the strongest; if A is independent of X , then their association is the weakest. However, neither context is useful because A does not provide additional information about X . In Section 5.2 , we propose a quantitative measurement of context usefulness that can be efficiently estimated and does not require knowledge of the downstream task. We also demonstrate empirically that the proposed metric correlates with the performance of the encoder on many real datasets. In one sentence, the key contribution of this work is clarity on the target of a large class of representation learning methods—the singular functions of the expectation operator. However, we do not discuss the numerical aspect of approximating these functions, which requires an expressive model architecture and a good optimizer, and such analyses are left to future work. 1.1 Examples of Representation Learning Methods Supervised learning is the simplest way to learn representations. For example, the representations of neural networks pretrained on ImageNet [ RDS+15 ] were very popular in the early days of the deep learning boom [ HAE16 ] . Specifically, one uses the output of an intermediate layer, typically the one before the last linear layer, as the representation. However, it has never been fully explained why the penultimate layer works so well on tasks very different from the pretraining task.

Read the complete text