Structured Contrastive Learning for Interpretable Latent Representations
Zhengyang Shen, Hua Tu, Mayue Shi · 2025
Read Structured Contrastive Learning for Interpretable Latent Representations by Zhengyang Shen, Hua Tu, Mayue Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14920.txt.
Introduction
Zhengyang Shen, Hua Tu, and Mayue Shi diagnose why neural networks are oddly fragile to changes that should not matter. A tiny time shift in a heart trace can wreck the match between two internal representations, and rotating a motion sensor can collapse activity recognition. The authors blame unconstrained learning: internal features drift freely as long as the final answer stays right.
Their Structured Contrastive Learning framework splits internal features into three groups by design. One group must stay fixed under known changes, one must actively track those changes, and one stays free for the task. The split gives builders direct control over what a representation should and should not notice.
Catalog record
| CREATOR | Zhengyang Shen, Hua Tu, Mayue Shi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-14920
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structured Contrastive Learning for Interpretable Latent Representations Zhengyang Shen, Hua Tu, Mayue Shi Abstract Neural networks exhibit severe brittleness to semantically irrelevant transformations. A mere 75ms electrocardiogram (ECG) phase shift degrades latent cosine similarity from 1.0 to 0.2, while sensor rotations collapse activity recognition performance with inertial measurement units (IMUs). We identify the root cause as “laissez-faire” representation learning, where latent spaces evolve unconstrained provided task performance is satisfied. We propose Structured Contrastive Learning (SCL) , a framework that partitions latent space representations into three semantic groups: invariant features that remain consistent under given transformations (e.g., phase shifts or rotations), variant features that actively differentiate transformations via a novel variant mechanism, and free features that preserve task flexibility. This creates controllable push-pull dynamics where different latent dimensions serve distinct, interpretable purposes. The variant mechanism enhances contrastive learning by encouraging variant features to differentiate within positive pairs, enabling simultaneous robustness and interpretability. Our approach requires no architectural modifications and integrates seamlessly into existing training pipelines. Experiments on ECG phase invariance and IMU rotation robustness demonstrate superior performance: ECG similarity improves from 0.25 to 0.91 under phase shifts, while WISDM activity recognition achieves 86.65% accuracy with 95.38% rotation consistency, consistently outperforming traditional data augmentation. This work represents a paradigm shift from reactive data augmentation to proactive structural learning, enabling interpretable latent representations in neural networks. 1 Introduction Neural networks often exhibit unexpected sensitivity to transformations that should be semantically irrelevant. This transformation brittleness manifests across domains: image classification systems fail under minor rotations ( Gao et al. 2020 ) , signal processing models become unreliable with temporal shifts ( Volpi & Murino 2019 ) , and activity recognition systems collapse under sensor orientation changes. As an example, this vulnerability became stark in similar ECG signal retrival for cardiac disease detection. Retrival based on variational autoencoders (VAEs) failed catastrophically when identical cardiac waveforms were temporally shifted ( Shen et al. 2025 ) . As shown in Figure 1 , even minor temporal shifts lead to significantly different latent representations, despite identical signal morphology. Cosine similarity drops rapidly from 1.0 to below 0.2, indicating that the learned representations unsuitable for clinical similarity retrival. Figure 1: Transformation brittleness in latent representations. Traditional VAEs exhibit severe phase sensitivity: identical ECG waveforms at different temporal positions produce dramatically different latent vectors, with cosine similarity degrading from 1.0 to below 0.2 across phase shifts (sampling rate: 400 Hz). This brittleness stems from “laissez-faire” representation learning —where learned representations evolve unconstrained as long as task performance is satisfied ( Zhao et al. 2022 ) . Traditional training objectives optimize solely for task performance while providing no explicit guidance on inter-sample relationships across transformations ( Higgins et al. 2018 ) . Information-theoretic analysis reveals that standard loss functions maximize mutual information between inputs and task-relevant features while remaining agnostic to semantic structure preservation ( Tian et al. 2020 ) . Networks can become excessively invariant to meaningful changes while remaining overly sensitive to irrelevant ones ( Jacobsen et al. 2018 ) , with fundamental impossibility results proving that invariance alone cannot identify meaningful latent structure ( Bing et al. 2023 ) . The standard response has been data augmentation—exposing models to transformed training data. Contrastive methods like SimCLR ( Chen et al. 2020 ) and MoCo ( He et al. 2020 ) achieve success by maximizing agreement between augmented views while minimizing agreement with other samples. However, these approaches typically apply uniform constraints across all latent dimensions and operate through discrete sampling of transformation space, providing no explicit control over representation structure. Recent advances identify critical limitations: forcing invariance to all augmentations can be suboptimal for downstream tasks ( Xiao et al. 2020 ) , with inherent trade-offs between accuracy and invariance that cannot be resolved through augmentation alone ( Zhao et al. 2022 ) . Data augmentation addresses symptoms rather than causes—increasing exposure to transformations without providing principled control over representation organization. We propose a paradigm shift from discrete data augmentation to structured latent space learning through explicit feature partitioning and contrastive control. Our framework transforms neural networks from black boxes into interpretable glass box systems with controllable semantic organization. The core innovation lies in partitioning latent representations into functionally distinct groups: invariant features that remain consistent across transformations, variant features that actively differentiate between transformations, and free features that remain unconstrained for task-specific optimization. Central to this structured contrastive learning is a contrastive objective that simultaneously pulls invariant features together while pushing features of different samples apart, enhanced through a variant mechanism that encourages variant features to differentiate even within positive pairs. This creates sophisticated learning where different latent aspects serve distinct, controllable purposes. Our approach builds upon disentanglement frameworks ( Higgins et al. 2018 ) , invariant causal representation learning ( Mitrovic et al. 2020 ) , and extends contrastive learning approaches like ContrastVAE ( Wang et al. 2022 ) and multi-level feature learning ( Xu et al. 2022 ) , while addressing fundamental limitations in purely invariance-based methods ( Bing et al. 2023 ; Zhao et al. 2022 ) . Our framework provides key advances: fine-grained control over different representation aspects through explicit partitioning, enhanced interpretability while maintaining task accuracy, and seamless integration into existing training pipelines without architectural modifications. We demonstrate effectiveness through comprehensive experiments on ECG phase invariance (improving cosine similarity from 0.25 to 0.91) and IMU rotation robustness (achieving 86.65% accuracy with 95.38% rotation consistency). Our contributions represent a paradigm shift from reactive data augmentation to proactive structural learning: (1) systematic identification of transformation brittleness as a universal neural network limitation; (2) a structured latent space learning framework transforming black box networks into interpretable glass box systems; (3) a structured contrastive learning mechanism with variant-enhanced control providing fine-grained feature organization; and (4) a practical training methodology enhancing interpretability and robustness without sacrificing performance. This opens new possibilities for deploying deep learning systems in critical applications requiring both performance and interpretability. 2 Method We present our structured contrastive learning framework that creates interpretable latent representations through explicit feature partitioning and controllable contrastive objectives. 2.1 Structured Latent Space Partitioning Traditional neural networks learn task-specific representations through purely task-oriented objectives: \mathcal{L}_{\text{traditional}}=\mathcal{L}_{\text{task}}(f(x),y) (1) where f(\cdot) is the encoder function and y is the task label. This allows latent representations to evolve freely, causing semantically similar inputs x and T(x) (where T represents transformations like phase shifts or rotations) to produce arbitrarily different representations: ||f(x)-f(T(x))|| can be large despite semantic equivalence. Figure 2: Structured contrastive learning in latent space. Top: Traditional data augmentation provides no control over latent representations. Bottom: Our method partitions features into invariant (pulled together), variant (pushed apart via variant mechanism), and free (unconstrained) components, transforming neural networks into interpretable systems with controllable semantic meaning. Our key insight is to explicitly model inter-sample relationships through structured contrastive learning: \mathcal{L}_{\text{structured}}=\mathcal{L}_{\text{task}}+\lambda\mathcal{L}_{\text{contrastive}} (2) We partition the latent representation f(x)\in\mathbb{R}^{d} into three semantically distinct subgroups: \displaystyle f(x) \displaystyle=[f_{\text{inv}}(x),f_{\text{var}}(x),f_{\text{free}}(x)] (3) where d_{\text{inv}}+d_{\text{var}}+d_{\text{free}}=d . Each subgroup serves a distinct purpose: Invariant features ( f_{\text{inv}} ) encode task-relevant information that should remain consistent across transformations; Variant features ( f_{\text{var}} ) capture transformation-specific information that should change predictably; Free features ( f_{\text{free}} ) learn unconstrained representations, preserving network flexibility. 2.2 Variant-Enhanced Structured Contrastive