UR–JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures

Triet M. Le, Ph.D · 2026

Read UR–JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures by Triet M. Le, Ph.D (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01443.txt.

Introduction

Le proposes UR-JEPA, a fix for collapse in predictive embedding models. Joint-embedding architectures can collapse to useless constants. LeJEPA fights collapse with an isotropic Gaussian target. That target clashes with the manifold hypothesis, though. Embeddings should sit on a low-dimensional subset, not spread evenly. UR-JEPA targets a uniformly rectifiable measure instead. A smoothed Carleson-type square function realizes that target.

On Inet10, UR-JEPA gains 0 point 83 points over LeJEPA. Seed variance drops by about 30 percent as well. Other datasets land in the same accuracy band at convergence. The geometric signature differs sharply, though. Its PCA spectrum drops four to five orders by index 20 to 25. LeJEPA's spectrum stays near flat in comparison.

Catalog record

CREATORTriet M. Le, Ph.D
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01443

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UR–JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures Triet M. Le, Ph.D Abstract A central difficulty in training Joint-Embedding Predictive Architectures (JEPAs) is preventing representation collapse. LeJEPA addresses this by enforcing an isotropic Gaussian target on the embeddings via Sketched Isotropic Gaussian Regularization (SIGReg). This target is in tension with the manifold hypothesis, which expects embeddings to concentrate on a low-dimensional subset of the ambient space. We propose UR–JEPA , which targets a uniformly n n -rectifiable measure of local tangent dimension n n at small scales, realized through a Gaussian-kernel smoothed Carleson-type square function ℒ CGLT \mathcal{L}^{\text{CGLT}} , with a complementary Jones β \beta -number formulation. On Inet10, UR–JEPA( ℒ CGLT \mathcal{L}^{\text{CGLT}} ) attains 0.9141 ± 0.0014 0.9141\pm 0.0014 for a + 0.83 +0.83 pp gain over LeJEPA( ℒ SIGReg \mathcal{L}^{\text{SIGReg}} ) with ∼ 30 % \sim 30\% lower seed standard deviation; on matched-recipe Galaxy10 SDSS, a single-seed ImageNet- 100 100 run, and a 3 3 -seed EuroSAT remote-sensing run, the two methods lie in the same peak-accuracy band at convergence, with UR–JEPA retaining its lower-seed-variance signature. On EuroSAT the in-domain pair is competitive at 96.0 96.0 to 96.1 % 96.1\% with large remote-sensing foundation-model transfer at a 25 × 25\times smaller backbone. The distinction is geometric: direct visualization of the projector output distribution shows that on all four datasets UR–JEPA( ℒ CGLT \mathcal{L}^{\text{CGLT}} ) produces a global PCA spectrum with a 4 4 to 5 5 order-of-magnitude drop at index ∼ 20 \sim 20 to 25 25 out of D = 32 D=32 , while LeJEPA’s spectrum is near-flat (top-to-bottom ratio at most 3.6 3.6 ). Per-dimension marginals are simultaneously near-Gaussian for both methods (mean Shapiro-Wilk W ∈ [ 0.992 , 0.996 ] W\in[0.992,0.996] ) as a Diaconis-Freedman consequence. At matched accuracy the two regularizers therefore yield structurally distinct projected representations. 1 Motivation Let f_{\theta}:\mathcal{X}\to\mathbb{R}^{D} be an encoder that maps inputs to embeddings. A Joint-Embedding Predictive Architecture (JEPA) learns \theta by predicting the embedding of one augmented view x^{\prime} of a datum from the embedding of another view x , in embedding space rather than in input (pixel) space. Denote by z=f_{\theta}(x) and z^{\prime}=f_{\theta}(x^{\prime}) the two embeddings, and let P be a (possibly identity) predictor. The predictive term is then \mathcal{L}_{\mathrm{pred}}(\theta)\;=\;\mathbb{E}_{(x,x^{\prime})}\!\bigl\|\,P(z)-\mathrm{sg}(z^{\prime})\,\bigr\|^{2}, (1) where \mathrm{sg}(\cdot) denotes a possible stop-gradient (removed in LeJEPA [ 1 ] ). Note that, on its own, ( 1 ) admits trivial solutions, for instance a constant f_{\theta} . The historical remedies for this collapse have been largely heuristic in design, in the sense that each was proposed to break a specific failure mode rather than derived from a target-distribution argument. They include the stop-gradient and EMA target encoders [ 9 , 10 , 11 ] , contrastive negatives [ 7 , 8 ] , the variance–covariance penalty of VICReg [ 14 ] , and the cross-correlation term of Barlow Twins [ 13 ] . LeJEPA replaces these heuristics. It identifies the optimal target distribution for z and penalizes deviation from it directly with a single explicit target-distribution regularizer (SIGReg), defined in Equation ( 2 ). Under the theoretical setting of [ 1 ] , this optimum is the isotropic standard Gaussian \mathcal{N}(0,I_{D}) . Manifold hypothesis. Requiring the embedding law to be isotropic Gaussian on all of \mathbb{R}^{D} forces the representation to spread across every direction of the ambient space. However, under the manifold hypothesis [ 21 , 22 ] we expect the embeddings of real data to concentrate on a subset of intrinsic dimension n\ll D . A more principled alternative would therefore shape the embedding distribution so that it appears quantitatively n -dimensional, rather than filling \mathbb{R}^{D} . Uniform n -rectifiability [ 33 , 35 ] is the canonical formulation of this intuition in geometric measure theory. Moreover, two complementary Carleson-type functionals characterize uniform n -rectifiability and are well-suited to an SGD-friendly loss: the CGLT theorem [ 24 ] , built on the scale-normalized ball density and its variations using smooth kernels, and the Jones \beta -number [ 25 , 33 , 34 ] , built on local affine n -plane approximation of embedding space. The paper is organized as follows. Section 2 surveys self-supervised methods related to UR–JEPA, with emphasis on the JEPA family and prior geometric or distributional regularizers. Section 3 reviews LeJEPA and its SIGReg loss, which we take as the closest comparable baseline. Section 4 introduces the geometric-measure-theoretic background, namely Ahlfors-David regularity and uniform n -rectifiability, and states the two characterizations (CGLT and Jones- \beta ) that our losses target. Section 5 develops the UR–JEPA technical design: the Gaussian-kernel smoothed Carleson loss family (§ 5.1 ) and the \beta -number loss via local PCA (§ 5.2 ), together with their anti-collapse mechanisms. Section 6 discusses degenerate optima, the numerical scale of each variant across CGLT forms, gradient noise, and distributed training. Section 7 reports the empirical study on ImageNet-10 (Inet10), Galaxy10 SDSS, and ImageNet-100 (Inet100), including the matched-recipe comparison against LeJEPA and the projector-geometry diagnostics (§ 7.6 ). Section 8 interprets the empirical findings in terms of what UR–JEPA recovers that LeJEPA cannot and what it sacrifices. Section 9 summarizes the contributions and Section 10 outlines future directions. 2 Related work The JEPA family. JEPAs cast self-supervised representation learning as the problem of predicting one view’s embedding from another’s, in the latent space rather than in pixels. Beginning with I–JEPA [ 2 ] , the family has grown to include video (V–JEPA [ 3 ] , V–JEPA 2 [ 4 ] ), joint motion–content learning (MC–JEPA [ 5 ] ), and various vision–language extensions [ 6 ] . Until LeJEPA, every published JEPA relied on at least one of the heuristics that prevent latent collapse, for instance EMA target encoders, stop-gradient, masking schemes that destroy trivial-solution paths, or hand-tuned learning-rate schedules. LeJEPA [ 1 ] is the first JEPA to discard this collection of heuristics in favor of SIGReg. UR–JEPA inherits the design philosophy of LeJEPA, but it adopts uniform rectifiability as the target regularization. Non-collapsing self-supervised learning, by mechanism. The broader self-supervised literature can be organized by its collapse-prevention mechanism (the following categorization is not exhaustive): 1. Contrastive losses with explicit negatives , e.g. SimCLR [ 7 ] and MoCo [ 8 ] , prevent collapse by pushing apart non-paired samples in a large in-batch dictionary. 2. Asymmetric architectures with stop-gradient and predictor heads , e.g. BYOL [ 9 ] , SimSiam [ 10 ] , DINO [ 11 ] , and DINOv2 [ 12 ] , avoid collapse via the dynamics of the predictor–EMA loop rather than via an explicit target. 3. Decorrelation / covariance-based losses , e.g. Barlow Twins [ 13 ] , VICReg [ 14 ] , and W–MSE [ 15 ] , push the empirical covariance toward the identity (full-rank, decorrelated marginals). 4. Hypersphere uniformity losses , e.g. Wang & Isola’s alignment–uniformity decomposition [ 16 ] , regularize L^{2} -normalized embeddings to spread uniformly on the unit sphere via a Gaussian-potential repulsion term. 5. Manifold-capacity losses , e.g. MMCR [ 17 ] , maximize a nuclear-norm proxy for the number of linearly separable view-manifolds, drawing on the statistical mechanics of high-dimensional classification. 6. Distributional matching , e.g. LeJEPA’s SIGReg [ 1 ]

Read the complete text