Structured Multidimensional Representation Learning for Large Language Models

A. Elichi, K. Jbilou, M. El Guide, F. Dufrenois · 2026

Read Structured Multidimensional Representation Learning for Large Language Models by A. Elichi, K. Jbilou, M. El Guide, F. Dufrenois (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05727.txt.

Introduction

El Ichi and team present a spectral splitting of vectors using the L-product for third-order tensors, tackling setting growth and repeat. Shaping tokens into spectral slices moves attention and feed-forward work into the transform domain. The Tensor Transformer splits the encoder into p spectral sub-transformers, keeping standard Transformer meaning. The L-Transformer equals p side-by-side Transformers on smaller vectors, yielding about 1/p encoder setting cut. Slice-set rate scaling improves spread. On IMDB and AG News, encoder settings fall up to 75% for p=4.

Catalog record

CREATORA. Elichi, K. Jbilou, M. El Guide, F. Dufrenois
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80854
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05727

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structured Multidimensional Representation Learning for Large Language Models A. Elichi, K. Jbilou, M. El Guide, F. Dufrenois Abstract Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks, but their scaling is accompanied by substantial parameter growth and redundancy in the embedding dimension. In this work, we introduce a structured spectral factorization of the embedding space based on the ℒ \mathcal{L} -product for third-order tensors. By reshaping token representations into spectral tensor slices and performing attention and feed-forward operations in the transform domain, we obtain a Tensor Transformer architecture that decomposes the encoder into p p independent spectral sub-transformers while preserving standard Transformer semantics. We prove that the proposed ℒ \mathcal{L} -Transformer is spectrally equivalent to p p parallel Transformers operating on reduced-dimensional embeddings, which yields an ≈ 1 / p \approx 1/p reduction (up to lower-order terms such as biases and normalization parameters) in encoder parameters under fixed total embedding size. When instantiated with a real-valued Discrete Cosine Transform (DCT), the method remains fully differentiable and compatible with existing training pipelines. Beyond compression, the spectral decomposition introduces an inductive bias over embedding frequencies, enabling slice-dependent frequency scaling that improves generalization. Experiments on IMDB and AG News show that the proposed model can substantially reduce encoder parameters (up to 75 % 75\% for p = 4 p{=}4 ) while maintaining competitive accuracy. On IMDB, the tensorized encoder matches or improves upon the standard baseline under compression, whereas on AG News at moderate width we observe a small accuracy decrease in exchange for a 4 × 4\times encoder reduction; at BERT-base width ( d = 768 d{=}768 ), performance returns to parity. keywords Tensor Transformer, \mathcal{L} -product, Transform-domain attention, Spectral tensor factorization, Discrete Cosine Transform (DCT), Structured representation learning, Parameter-efficient Transformers 1 Introduction Transformer architectures [ 33 ] have become the dominant paradigm in modern pattern recognition and natural language processing, achieving state-of-the-art performance in text classification, machine translation, and large-scale language modeling [ 2 , 31 , 5 , 16 , 24 ] . However, their success is accompanied by substantial growth in parameter count, particularly in the embedding dimension and feed-forward layers. As model width increases, redundancy and overparameterization emerge, motivating research on structured compression and low-rank approximations [ 35 , 27 , 18 ] , including distillation [ 25 , 8 ] , architectural parameter sharing [ 14 ] , and structured factorizations [ 30 ] . Most existing compression strategies operate directly in the weight space through pruning, low-rank matrix factorization, or post-hoc rank reduction. While effective, these approaches approximate a pre-trained model and do not fundamentally alter the representation geometry of the embedding space. An alternative perspective consists in reparameterizing the embedding dimension itself through structured spectral factorization, thereby imposing algebraic structure prior to training. Tensor algebra provides a natural framework for structured multidimensional representations [ 13 , 23 , 11 ] . In particular, the \mathcal{L} -product for third-order tensors defines multiplication via an invertible linear transform applied along a designated mode [ 9 ] . In the transform domain, this operation induces a block-diagonal structure that decouples tensor slices while preserving global consistency under inverse transformation. In this work, we leverage this algebraic property to introduce a Tensor Transformer architecture. Token embeddings are reshaped into third-order tensors, and attention as well as feed-forward operations are reformulated using the \mathcal{L} -product. Our main theoretical result shows that the resulting encoder is spectrally equivalent to p independent Transformers operating on reduced-dimensional embeddings in the transform domain. This equivalence yields an \approx 1/p reduction in encoder parameters while maintaining the expressive capacity of standard self-attention within each spectral slice. The proposed construction is not equivalent to a simple partition of the embedding dimension. The forward and inverse transforms couple spectral components through a global linear operator, allowing structured mixing across slices after each attention and feed-forward update. When instantiated with the Discrete Cosine Transform [ 1 ] , the framework remains real-valued, fully differentiable, and compatible with standard optimization pipelines. In addition to parameter reduction, the spectral formulation introduces a frequency-domain inductive bias. By assigning slice-dependent positional scaling coefficients, the model can emphasize low-frequency components or distribute attention harmonically across spectral channels. This structured spectral weighting can help task-dependent and stability in moderate-scale classification tasks. Empirical evaluations on IMDB and AG News demonstrate that the Tensor Transformer substantially reduce encoder parameters. In certain configurations, spectral weighting further improves predictive performance relative to standard Transformer baselines. These findings suggest that structured spectral factorization provides a principled and computationally efficient alternative to flat embedding representations in attention-based models. Section 2 reviews related work on efficient Transformers, low-rank adaptation, tensorized parameterizations, and transform-based mixing. Section 3 introduces tensor notation and the algebraic operations needed in the sequel. Section 4 defines the \mathcal{L} -transform and \mathcal{L} -product, and recalls key properties such as slice-wise multiplication and the \mathcal{L} -SVD. Section 5 presents the proposed tensorized Transformer architecture: the embedding tensorization, \mathcal{L} -multi-head attention, \mathcal{L} -FFN, normalization, and the slice-wise equivalence result underpinning the \approx 1/p encoder-parameter scaling. Section 6 reports empirical results and ablations on IMDB and AG News, including parameter-matched baselines, scaling to d=768 , and efficiency measurements. Finally, Section 7 concludes and outlines limitations and future directions, while the appendix provides additional implementation details and supplementary analyses. 2 Related Work Transformers have become the dominant architecture for sequence modelling by combining multi-head self-attention with position-wise feed-forward blocks, enabling scalable representation learning across language, vision, and multimodal settings [ 33 ] . Scaling this architecture has also produced foundation language models that achieve strong few-shot performance and broad generalization [ 2 , 31 , 5 , 16 , 24 ] . At the same time, standard self-attention can be memory intensive and exhibits structural redundancies (including redundancy across heads), motivating a large body of work on approximating token interactions and compressing parameters while preserving downstream accuracy [ 35 , 12 , 34 , 3 , 4 ] . These efforts range from architectural modifications that alter how tokens interact, to post-training interventions that directly modify weight matrices, as well as parameter-efficient fine-tuning strategies that avoid updating the full model. A prominent family of approaches targets redundancy through low-rank structure. In parameter-efficient fine-tuning, low-rank adapters inject trainable rank-constrained updates into pretrained linear maps, yielding strong accuracy at low update cost [ 7 ] . Complementary post-training methods compress or modify pretrained weights without additional training. Layer-selective rank reduction removes higher-order components of selected weight matrices and can improve downstream reasoning without adding parameters or requiring extra data [ 27 ] . Related tensor-centric compression ideas can also be applied directly to collections of matrices arising in large models. TRAWL represents model parameters in a tensor format and performs structured reduction to lower storage and computation costs [ 19 ] . Together, these works motivate treating groups of related matrices as structured objects rather than independent 2D arrays, which is especially natural in multi-head attention where heads form an additional mode with meaningful correlations. Complementary compression lines include knowledge distillation for pretrained Transformers [ 25 , 8 ] , architectural parameter sharing [ 14 ] , and structured matrix factorizations tailored to Transformer blocks [

Read the complete text