MIPIC : Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chaining

Phung Gia Huy, Hai An Vu, Minh-Phuc Truong, Thang Duc Tran, Linh Ngo Van, Thanh Nguyen Hong, Trung Le · 2026

Read MIPIC : Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chaining by Phung Gia Huy, Hai An Vu, Minh-Phuc Truong, Thang Duc Tran, Linh Ngo Van, Thanh Nguyen Hong, Trung Le (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24374.txt.

Introduction

Phung and six colleagues present MIPIC, a training method producing nested Matryoshka embeddings that stay useful at compute budgets. Nested embeddings allow flexible inference, but arranging data across size and depth needs care. MIPIC lines up token-level geometric and attention links between full and short forms through self-taught alignment. It also chains task meanings from deeper layers into earlier ones step by step. Tests on STS, NLI, and classification tasks, spanning TinyBERT to BGEM3 and Qwen3, show strong results at all sizes, with clear gains at tiny sizes.

Catalog record

CREATORPhung Gia Huy, Hai An Vu, Minh-Phuc Truong, Thang Duc Tran, Linh Ngo Van, Thanh Nguyen Hong, Trung Le
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58196
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24374

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MIPIC : Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chaining Phung Gia Huy, Hai An Vu, Minh-Phuc Truong, Thang Duc Tran, Linh Ngo Van, Thanh Nguyen Hong, Trung Le Abstract Representation learning is fundamental to NLP, but building embeddings that work well at different computational budgets is challenging. Matryoshka Representation Learning (MRL) offers a flexible inference paradigm through nested embeddings; however, learning such structures requires explicit coordination of how information is arranged across embedding dimensionality and model depth. In this work, we propose MIPIC (Matryoshka Representation Learning via Self-Distilled Intra-Relational Alignment and Progressive Information Chaining), a unified training framework designed to produce structurally coherent and semantically compact Matryoshka representations. MIPIC promotes cross-dimensional structural consistency through Self-Distilled Intra-Relational Alignment (SIA), which aligns token-level geometric and attention-driven relations between full and truncated representations using top-k CKA self-distillation. Complementarily, it enables depth-wise semantic consolidation via Progressive Information Chaining (PIC), a scaffolded alignment strategy that incrementally transfers mature task semantics from deeper layers into earlier layers. Extensive experiments on STS, NLI, and classification benchmarks (spanning models from TinyBERT to BGEM3, Qwen3) demonstrate that MIPIC yields Matryoshka representations that are highly competitive across all capacities, with significant performance advantages observed under extreme low-dimensional. 2 2 footnotetext: Corresponding author: [email protected] 1 Introduction Learned dense representations are the cornerstone of modern NLP ( Mikolov et al. 2013 ; Pennington et al. 2014 ; Devlin et al. 2019 ) , yet their high computational demands often limit deployment ( Wang et al. 2020 ) . Matryoshka Representation Learning (MRL) ( Kusupati et al. 2022 ) addresses this by introducing nested embeddings that enable adaptive inference-time truncation, allowing a single model to satisfy diverse computational budgets without retraining. Training effective Matryoshka representations, however, presents challenges that extend beyond conventional embedding learning. Unlike standard representations, MRL requires semantic information to be explicitly organized such that meaningful structure is preserved under progressive dimensional truncation. Existing MRL formulations ( Kusupati et al. 2022 ; LI et al. 2025 ) primarily enforce nested usability through independent supervision at different truncation levels or sentence-level alignment objectives. While such strategies encourage prefix usability, they leave open the broader question of how semantic information should be internally arranged to support robust Matryoshka structures. In particular, learning low-dimensional prefix embeddings that remain semantically expressive requires the model to carefully organize semantic features across both embedding dimensions and network layers, involving a non-trivial restructuring of how information is represented and propagated. First, effective MRL can be viewed through the lens of cross-dimensional structural alignment , where low-dimensional prefixes are encouraged to reflect not only sentence-level outputs but also the internal relational structure (like token-level geometric relationships, hidden states) of the full-dimensional representations. From this perspective, aligning relational patterns across different dimension spaces provides a natural way to support stable semantic behavior in compact prefixes. Second, MRL may also be understood from the perspective of involving depth-wise semantic consolidation , in which task-relevant information is gradually condensed from deeper layers, where representations are typically more expressive, into earlier layers. This process is inherently progressive: rather than emerging at a single truncation point, core semantic features can be refined and transferred as information flows through the network. Viewing Matryoshka training through this lens highlights the benefits of intermediate guidance that supports a smooth, coarse-to-fine refinement of representations across depth. Motivated by these considerations, we propose MIPIC ( M atryoshka Representation Learning via Self-Distilled I ntra-Relational Alignment and P rogressive I nformation C haining), a unified training framework that explicitly organizes information across both embedding dimensionality and model depth, enabling the learning of structurally coherent and semantically compact Matryoshka representations. To encourage cross-dimensional structural consistency, MIPIC employs Self-Distilled Intra-Relational Alignment (SIA). Rather than aligning representations solely at the sentence level like prior MRL approaches ( Kusupati et al. 2022 ; LI et al. 2025 ) , SIA focuses on token-level intra-relations derived from geometric similarity and attention-based importance. Using Centered Kernel Alignment (CKA) ( Kornblith et al. 2019 ) with a hard top- k selection strategy, SIA selectively aligns the most salient relational patterns between full and truncated representations. This targeted self-distillation goes beyond simple alignment; it effectively reorganizes the internal feature hierarchy, forcing critical semantic structures to be prioritized and concentrated within the low-dimensional prefix dimensions. Complementing cross dimensional alignment, MIPIC introduces Progressive Information Chaining (PIC) to guide the flow of semantic information across the network depth. Unlike prior methods that restrict task supervision solely to the final representation, which can be unstable when compressing rich knowledge into narrow dimensions, PIC builds a scaffolded alignment pipeline in which each step acts as a semantic bridge. This design is motivated by empirical findings that lower Transformer layers predominantly encode low level linguistic features such as syntax and basic semantics, while higher layers increasingly capture task specific and discriminative signals ( Jawahar et al. 2019 ; Liu et al. 2019 ) . By propagating task aware cues from upper layers into earlier ones, PIC performs a controlled early intervention that gently biases the formation of low level representations toward task relevant subspaces, ensuring that useful discriminative structure is available from the beginning of the network processing. Importantly, by aligning only the most essential components of each layer representation, which have been reorganized by SIA to concentrate the most critical task relevant knowledge into a compact subspace, PIC allows the remaining dimensions to continue modeling low level and general linguistic features. This selective supervision preserves the model natural representational flexibility, avoids over regularization, and enables a stable and progressive refinement of task relevant information across depth. Together, SIA and PIC provide a principled training framework for MRL that explicitly organizes semantic information across both dimensionality and depth. Our main contributions are: • We propose MIPIC , a unified framework that addresses structural and semantic incoherence in MRL by explicitly coordinating cross-dimensional alignment and depth-wise information consolidation. • We introduce two complementary mechanisms: Self-Distilled Intra-Relational Alignment (SIA), which utilizes top-k CKA self distillation to enforce cross-dimensional topological consistency via self-distillation, and Progressive Information Chaining (PIC), a scaffolded learning strategy that enables lower-dimensional representations to acquire task-relevant information early in training through progressive semantic guidance across network depth. • We present extensive experiments on STS, NLI, and text classification benchmarks showing that MIPIC significantly improves representation efficiency across diverse backbones, from TinyBERT-6L and BERT-base to large-scale models such as Qwen3 embedding 0.6B and BGE-M3. MIPIC remains competitive at full capacity while consistently outperforming state-of-the-art baselines under severe truncation, particularly in low-dimensional regimes. 2 Related Work and Background 2.1 Related Works Learning Sentence Embeddings Sentence embeddings have evolved from static word vectors ( Pennington et al. 2014 ; Mikolov et al. 2013 ) and contextual models ( Peters et al. 2018 ; Devlin et al. 2019 ) , which initially lacked optimized sentence-level semantics. Sentence-BERT ( Reimers and Gurevych 2019 ) addressed this via siamese fine-tuning, dramatically improving retrieval efficiency. Subsequent work introduced contrastive learning ( Gao et al. 2022 ; Nishikawa et al. 2022 ) to enhance robustness, while recent advances leverage Large Language Models to further refine representation quality ( He et al. 2025 ; BehnamGhader et al. 2024 ; Tao et al. 2025 ; Li and Zhou 2025 ) . More

Read the complete text