Learn from your own latents and not from tokens: A sample-complexity theory

Daniel J. Korchinski, Alessandro Favero, Matthieu Wyart · 2026

Read Learn from your own latents and not from tokens: A sample-complexity theory by Daniel J. Korchinski, Alessandro Favero, Matthieu Wyart (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27734.txt.

Introduction

Daniel Korchinski, Alessandro Favero, and Matthieu Wyart prove that predicting one's own latents beats token-level learning on sample complexity. For grammar data with hidden trees of depth L, token-level methods need samples exponential in L, while latent prediction needs only a constant number up to logarithmic factors. The bound is confirmed with hierarchical clustering, an end-to-end network, and the first sample-complexity analysis of data2vec.

Catalog record

CREATORDaniel J. Korchinski, Alessandro Favero, Matthieu Wyart
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77505
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27734

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learn from your own latents and not from tokens: A sample-complexity theory Daniel J. Korchinski, Alessandro Favero, Matthieu Wyart Abstract Generative models, from diffusion models to large language models, achieve remarkable performance but at a cost in training data orders of magnitude larger than what biological learners require. An alternative paradigm has emerged in which networks are trained to predict their own latent representations of related views or masked regions, as in data2vec and JEPA – an idea related to predictive-coding accounts of the cortex. Despite strong empirical results, the theoretical understanding of these methods remains limited. Central questions include: by how much does latent prediction actually improve data efficiency? Is there a benefit to stacking such methods into multi-scale hierarchies? We answer both using as data a tractable probabilistic context-free grammar that captures the compositional structure of natural language and images. Such a grammar generates strings of visible tokens by recursively applying production rules along a tree of hidden symbols of depth L L . For such data, supervised or token-level SSL require a number of samples exponential in L L to recover the latent tree; we prove that latent prediction achieves this with a number of samples constant in L L , up to logarithmic factors. We confirm this bound with (i) a hierarchical clustering algorithm, (ii) an end-to-end neural network whose predictor-clusterer modules predict their own latents at each level via gradient descent, and (iii) the first sample-complexity analysis of data2vec, which we show implicitly performs hierarchical latent prediction. This suggests that explicit stacking such as H-JEPA is largely redundant. 1 Introduction Generative models have reached striking empirical success. Diffusion models produce realistic images and video [ 1 , 2 , 3 ] , while large language models trained by next-token prediction master grammar, world knowledge and reasoning [ 4 , 5 , 6 ] . Both rest on a single recipe: predict masked or future fragments of the raw signal at massive scale. Yet this success is bought at a cost biological learners do not pay. Frontier LLMs are trained on 10^{13} – 10^{14} tokens [ 7 , 8 ] , more than five orders of magnitude beyond what a child encounters before reaching adult-level competence [ 9 , 10 , 11 ] ; state-of-the-art diffusion models likewise rely on billions of images [ 12 ] . This gap suggests that token-level pretraining is far from optimal, and several hypotheses have been advanced to explain it. One is that biological learning is multimodal and grounded [ 13 , 14 ] . A second – the one we pursue – is that learning may not be most efficient at the level of raw tokens, but instead in a more abstract latent space [ 15 , 16 , 17 ] . Indeed, predicting semantic concepts or continuous latents has been shown to improve data efficiency [ 18 , 19 ] . An alternative strategy to input reconstruction has gained traction in the last five years: the network is trained to predict its own latent representations of related views, masked regions or future signals. These targets are themselves the output of a learned encoder, lifting prediction into increasingly abstract spaces as learning progresses. The idea relates to computational neuroscience, where predictive coding posits that the cortex seeks to predict its own future activity [ 15 , 20 , 21 ] . A growing family of self-supervised methods instantiates this principle, including BYOL [ 22 ] , DINO [ 23 ] , data2vec [ 24 ] and JEPA [ 17 ] . Some advantages of these algorithms over input reconstruction approaches have been discussed in the literature, including the fact that they can be formulated as energy-based methods [ 17 ] or are more robust to the presence of irrelevant features in the data [ 25 ] . Yet understanding why and by how much these methods improve data efficiency remains a fundamental challenge in both machine learning and neuroscience. Another key question is the putative benefits of stacking such algorithms so as to obtain a multi-scale hierarchy of learned representations. This idea, proposed years ago [ 17 ] , was recently implemented [ 26 , 27 ] and appears to lead at best to moderate improvements, for reasons that remain currently unclear. In this work, we address both questions by focusing on synthetic models of data with a hierarchical latent structure. Probabilistic context-free grammars (PCFGs) are a natural candidate: through recursive production rules they generate data compositionally from a tree of latent variables, and have long been influential as models of language [ 28 , 29 ] and natural images [ 30 , 31 ] . General PCFGs are notoriously hard to treat. We therefore work with the Random Hierarchy Model (RHM) [ 32 ] , a simplified PCFG with fixed tree topology and random production rules. The RHM has produced quantitatively predictive theories, including relating neural scaling laws to the statistics of natural language [ 33 , 34 ] , and a theory of compositional generalization and memorization in diffusion models for language and images [ 35 , 36 , 37 ] . The RHM is parameterized by the number of production rules per symbol, m , and the depth of the latent tree, L . Prior work shows that for deep architectures, supervised end-to-end learning requires an order of m^{L} samples [ 32 ] , and token-level self-supervised objectives – masked-language modelling and diffusion – require of order m^{L+1} [ 33 , 38 , 39 ] . Both costs grow exponentially with the depth L of the latent hierarchy (and thus polynomially in the input dimension, i.e., sequence length). Against this baseline, we establish three results: 1. Efficient algorithm for representation learning ( Section 3 ) . We show that a hierarchical clustering algorithm recovers the full non-root latent tree from order m^{3} samples – independent of the depth L , and hence exponentially fewer than those required by token-level SSL. 2. An end-to-end stacked architecture with the same scaling ( Section 4 ) . We propose a new architecture where a stack of small predictor-clusterer modules predicts its own latents at each level. Trained end-to-end with gradient descent, it reaches the m^{3} scaling on the RHM. 3. A theory of data2vec ( Section 5 ) . We show empirically and explain theoretically with reasonable assumptions that data2vec [ 24 ] implicitly performs hierarchical latent prediction, again leading to a sample complexity of order m^{3} . To our knowledge, this is the first sample-complexity analysis of such methods. These results establish that learning from one’s own latents can yield huge sample-complexity gains over token-level objectives. They also show that a single latent-prediction module such as data2vec is in effect already performing multi-scale latent discovery. Such methods may already be implicitly hierarchical, weakening the case for explicit stacking such as H-JEPA. Related work. The compositional and hierarchical structure of natural data has long been put forward as a key feature making it learnable by neural networks. Deep networks represent hierarchically compositional functions with exponentially fewer parameters than shallow ones [ 40 ] , yielding nonparametric sample-complexity bounds polynomial in the input dimension for both supervised [ 41 ]

Read the complete text