NdLinear: Don’t Flatten! Building Superior Neural Architectures by Preserving N-D Structure
Alex Reneau, Jerry Yao-Chieh Hu, Zhongfang Zhuang, Ting-Chun Liu, Xiang He, Judah Goldfeder, Nadav Timor, Allen G Roush, Ravid Shwartz-Ziv · 2025
Read NdLinear: Don’t Flatten! Building Superior Neural Architectures by Preserving N-D Structure by Alex Reneau, Jerry Yao-Chieh Hu, Zhongfang Zhuang, Ting-Chun Liu, Xiang He, Judah Goldfeder, Nadav Timor, Allen G Roush, Ravid Shwartz-Ziv (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17353.txt.
Introduction
Alex Reneau and co-authors introduce in 2025 NdLinear, a layer that works on tensors in their native shape. Standard layers flatten multi-axis inputs, which wastes the structure inside them. NdLinear instead transforms along each axis in turn, with no flattening step.
Parameter counts fall by orders of magnitude in many builds. Theory ties the design to structured decomposition while keeping learning power. A low-rank variant matches standard adapters on language tasks with far fewer weights. Trials span vision, language, time series, and table tasks with steady efficiency gains. Known limits appear on tasks where distant axes tangle tightly together.
Catalog record
| CREATOR | Alex Reneau, Jerry Yao-Chieh Hu, Zhongfang Zhuang, Ting-Chun Liu, Xiang He, Judah Goldfeder, Nadav Timor, Allen G Roush, Ravid Shwartz-Ziv |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-17353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NdLinear: Don’t Flatten! Building Superior Neural Architectures by Preserving N-D Structure Alex Reneau, Jerry Yao-Chieh Hu, Zhongfang Zhuang, Ting-Chun Liu, Xiang He, Judah Goldfeder, Nadav Timor, Allen G Roush, Ravid Shwartz-Ziv Abstract Many high-impact machine learning tasks involve multi-dimensional data such as images, volumetric medical scans, and multivariate time-series. Yet, most neural architectures flatten these inputs, discarding critical cross-dimension information. We introduce NdLinear , a novel linear transformation that circumvents this destructive flattening by operating directly on tensors. NdLinear applies transformations separately along each data dimension, thereby preserving the native data structure. Extensive experiments demonstrate NdLinear’s capacity to significantly enhance representational power, achieve dramatic parameter reductions (often by orders of magnitude), and maintain a favorable computational profile. For instance, when applied to Large Language Model finetuning, our NdLinear-LoRA delivers comparable or improved accuracy on reasoning tasks using up to 9 × 9\times fewer trainable parameters than standard LoRA. These broad advantages of NdLinear are consistently validated across diverse neural architectures (CNNs, RNNs, Transformers, MLPs) and data domains, including vision, language, time-series, and tabular tasks. As a versatile, drop-in replacement for standard linear layers, NdLinear processes data in its original N-dimensional form, offering a foundational component for developing more efficient and powerful next-generation neural architectures. † † footnotetext: NdLinear is licensed by Ensemble AI under the Apache 2.0 license. Our technical contributions and research findings are free for use by the open-source community. 1 Introduction Effectively processing N-dimensional (N-D) tensors is critical in modern artificial intelligence, with applications spanning volumetric medical scans ( Milletari et al. 2016 ) , spatio-temporal video, multivariate time series ( Liu et al. 2023 ) , and outputs of complex simulations ( Shanahan et al. 2023 ; Brown et al. 2020 ; Dosovitskiy et al. 2021 ; Radford et al. 2021 ) . However, a core challenge remains: fully retaining and leveraging the rich, inherent structure of N-D data within neural architectures. While specialized layers like convolutions excel at capturing local spatial hierarchies (e.g., in 2D/3D images), they are not universally optimal, especially for comprehensively propagating information in higher-dimensional data (e.g., >3D) without excessive complexity. Fundamentally, many neural architectures (CNNs, Transformers, MLPs and RNNs) still rely on standard Linear (fully connected) layers, which expect 1D vector inputs. This common practice necessitates flattening multi-dimensional feature maps or tensors before these transformations, for instance, in network heads, channel-wise projections, or MLP blocks within attention mechanisms. This ubiquitous flattening, even if localized, is often detrimental. It discards the data’s explicit N-D organization, disrupting critical spatial, temporal, or other relationships encoded across dimensions ( Sokar et al. 2024 ) . For example, valuable spatial hierarchies preserved by convolutions can be lost when feature vectors are decontextualized by flattening. In multivariate time series (e.g., dimensions for time, sensors, features), flattening intermingles these distinct axes, forcing models to rediscover temporal patterns or inter-sensor dependencies without structural priors. This loss of explicit structure forces networks to relearn intrinsic dependencies, potentially increasing parameter counts, data requirements, and computational load, while risking spurious correlations. To address this, we introduce NdLinear , a novel linear layer that operates natively on N-D tensors. Unlike standard linear layers, NdLinear processes data in its original N-D form by applying distinct linear transformations sequentially along each tensor dimension. This dimension-wise approach inherently preserves the data’s N-D structure. Preserving data’s native organization allows NdLinear to offer compelling advantages: It enhances representational power by maintaining structural integrity and facilitating a more natural information flow. It dramatically reduces parameter counts (often by orders of magnitude vs. flattened layers) and ensures efficient computation without sacrificing performance. These efficiencies lead to faster training/inference and lower memory usage. As a general-purpose layer, NdLinear is a versatile, drop-in replacement for standard linear layers in many architectures, benefiting diverse domains by enabling models to better leverage N-D data characteristics. Our work makes the following primary contributions, substantiated by extensive empirical validation: • We articulate the critical limitations of flattening in conventional linear layers for N-D data. • We introduce and formulate NdLinear , a novel N-D linear layer using sequential, dimension-wise transformations to preserve data structure and achieve substantial parameter efficiency. • We empirically demonstrate through comprehensive evaluations that NdLinear significantly enhances model performance, or matches it with drastically fewer parameters, when replacing standard linear layers across diverse architectures (CNNs, RNNs, Transformers, MLPs) and data domains, including vision, language, time-series, and tabular data. • We analyze the significant parameter and computational efficiency gains originating from NdLinear’s dimension-wise transformation principle. • We establish NdLinear as a versatile and foundational building block for developing more efficient, structurally-informed, and powerful neural architectures. The remainder of this paper is organized as follows. Section 2 details the NdLinear transformation. Section 3 analyzes its parameter efficiency, computational complexity, and representational capacity. Section 4 presents experimental results demonstrating NdLinear’s efficacy. Finally, Section 5 summarizes findings, discusses implications, and outlines future work. 2 NdLinear: N-dimensional Linear Transformation This section formally introduces the NdLinear transformation. We first define the general problem of transforming tensor-structured data. Then, we detail the NdLinear layer, including its operational mechanics and algorithm. Finally, we situate NdLinear within the context of related methods. 2.1 Problem Formulation This work addresses learnable linear transformations for N-dimensional (N-D) input tensors. Such an input, X\in\mathbb{R}^{B\times D_{1}\times\cdots\times D_{N}} , comprises a batch of B samples, each with N feature dimensions of sizes D_{1},\ldots,D_{N} (e.g., height, width, channels for images; sequence length, embedding dimensions for text). The objective is to transform X into an output tensor Y^{\prime}\in\mathbb{R}^{B\times H_{1}\times\cdots\times H_{N}} , where each k -th feature dimension is reshaped from its original size D_{k} to a new size H_{k} . Our focus is specifically on the formulation of this N-D linear transformation. A primary challenge is that conventional linear layers necessitate flattening X into a 2D matrix (batch size \times total features). This operation discards the tensor’s explicit N-D structure. We aim to define a linear transformation that operates directly on N-D tensors, thereby preserving their inherent multi-dimensional organization. 2.2 The NdLinear Transformation The NdLinear transformation processes N-D tensors by applying separate linear transformations along each of their feature dimensions sequentially. This contrasts with standard linear layers, which flatten the N-D tensor into a 2D matrix, losing the original multi-dimensional structure. By operating on each dimension independently while preserving the others, NdLinear retains the inherent structural information of the data throughout the transformation. Conceptually, for an input tensor X\in\mathbb{R}^{B\times D_{1}\times\cdots\times D_{N}} , NdLinear learns N distinct weight matrices W_{1},\ldots,W_{N} , where each W_{k}\in\mathbb{R}^{D_{k}\times H_{k}} transforms the k -th dimension of the input from its original size D_{k} to a new size H_{k} , with optional bias vectors b_{k}\in\mathbb{R}^{H_{k}} per dimension. The transformation is applied iteratively: the output from transforming dimension k becomes the input for transforming dimension k+1 . The procedure is detailed in Algorithm 1 . Algorithm 1 NdLinear Transformation 0: Input tensor X\in\mathbb{R}^{B\times D_{1}\times\cdots\times D_{N}} (batch size B , original feature dimensions D_{1},\ldots,D_{N} ), Target output feature dimensions H_{1},\ldots,H_{N} , Learnable weight matrices W_{k}\in\mathbb{R}^{D_{k}\times H_{k}} for k=1,\ldots,N , Optional learnable bias vectors b_{k}\in\mathbb{R}^{H_{k}} for k=1,\ldots,N . Let X_{\text{out}}\leftarrow X . for k=1 to N do Let current shape of X_{\text{out}} be (B,S_{1},\ldots,S_{N}) , where S_{j}=H_{j} for j<k , and S_{j}=D_{j} else. The k -th feature dimension (original size