On Dimension-Free Transformer: An Application of STP to AI
Daizhan Cheng · 2025
Read On Dimension-Free Transformer: An Application of STP to AI by Daizhan Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14514.txt.
Introduction
Daizhan Cheng proposes a transformer that drops fixed input size in a solo 2025 paper. Transformers, the models behind much of modern AI, normally need all inputs to share one set size. His tool for lifting that limit is the semi tensor product, a way to multiply matrices of mismatched shapes.
He first writes each part of a transformer in matrix form. Then he swaps each linear step for a projection based step that works on vectors of any size. The new frame lets inputs and outputs vary in length from case to case. Cheng argues that using facts from all entries at once should make the model more efficient with signals.
Catalog record
| CREATOR | Daizhan Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14514
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Dimension-Free Transformer: An Application of STP to AI Daizhan Cheng Abstract The matrix expressions for every parts of a transformer are firstly described. Based on semi-tensor product (STP) of matrices the hypervectors are reconsidered and the linear transformation over hypervectors is constructed by using projection. Its properties and calculating formulas are obtained. Using projection-based transformation of hypervector (PBTH), the framework of dimension-free transformer (DFT) is proposed by verifying each linear transformation in a transformer and replacing it by a proper PBTH, which allows the inputs and outputs being of arbitrary dimensions. Using balanced information about all entries, DFT must be more efficient in dealing with signals. I Introduction In recent years the development of AI is unbelievably fast, and its impact on human lives is amazingly increasing. Particularly, the ChartGPT, the DeepSeek etc., as excellent examples of connectionism in AI, are shacking the AI community as well as the human society. The key engine in ChartGPT and DeepSeek etc. is the transformer, which is an attention-based neural network [ 18 ] . Attentions are the key in a transformer. Fig. 1 , redrawing from [ 19 ] , describes it. [formula] [formula] [formula] [formula] Layer(Attention) [formula] Layer(Attention) Layer(Attention) [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] Fig. 1: Attenders in a Transformer From mathematical point of view, an attention is a multi-linear mapping \displaystyle\pi:x^{1}\times\cdots\times x^{s}\rightarrow y^{1}\times\cdots y^{t}. (1) • Case 1, Equal Length Sequences: \dim(x_{i})=\dim(y_{j}) , i\in[1,s] , j\in[1,t] . Then the mappings \pi are commonly described by matrix multiplication (MatMul), or linear mapping (Linear) in component-wise. • Case 2, Length-Varying Sequence: Say, \dim{x_{i}}=m_{i} , i\in[1,s] , \dim(y_{j})=n_{j} , j\in[1,t] . In literature the most commonly used technique to deal with this case is using zero-padding. Than is, assume m_{0}=\max_{1\leq i\leq s}m_{i} and n_{0}=\max_{1\leq j\leq t}n_{j} ), then we pad x^{i} by \tilde{x}^{i}=(x^{i},\underbrace{0,\cdots,0}_{m_{0}-m_{i}})^{\mathrm{T}z},\quad i\in[1,s]. Then the transformation defined in Case 1 can be used to deal with the padded date. After the transformation, we received \tilde{y}_{j} with length n_{0} . Then the so-called Padding-Mask is used to truncate it back to y_{j}\in{\mathbb{R}}^{n_{j}} . The zero-padding approach has several weaknesses. (a) The additional zero-padded 0 will cause some junk information into the resulting data. Then a padding mask is necessary to reduce the effects caused by such junk information. (b) The components of a vector are not used in a balance way. There are some biases. (c) The nominal m_{0} and n_{0} must be greater than or equal to the maximum m_{i} , i\in[1,s] or maximum n_{j} , j\in[1,t] respectively. Then the padded \tilde{x}^{i} , i\in[1,s] (similarly, \tilde{y}^{j} , j\in[1,t] ) together become a sparse matrix, which also wast some memories. This paper investigates the process on length-varying sequences. Instead of zero-padding approach, a projection-based padding is proposed, which can overcome the above mentioned weaknesses of zero-padding. Based on projection-based padding, a new transformer, called the dimension-free transformer (DFT) is proposed. In DFT, all kinds of linear transformations in a transformer have been reconsidered and replaced by corresponding “dimension-free” ones, which are essentially the projection-based padding and un-padding. In the sequel, you will see that the projection-based padding is also available for causal mask. Next, let us observe the whole structure of a transformer depicted by Fig. 2 , which is redrawn from [ 18 ] . [formula] [formula] [formula] [formula] Add & Norm Add & Norm [formula] [formula] Add & Norm Add & Norm [formula] Add & Norm Linear [formula] [formula] [formula] [formula] [formula] [formula] Inputs [formula] [formula] [formula] [formula] [formula] Fig. 2: A Transformer There are only few nonlinear functions, which are: \begin{array}[]{l}softmax,\quad(\mbox{probabilitization}),\\ Norm,\quad(\mbox{normalization}),\\ ReLU,\quad(\mbox{rectified linear Unit}),~\mbox{etc.}\end{array} All of them can be considered as “self-adjustments” of argument vectors/matrices without any dimension change, where self-adjustment means no other data are necessary. Hence there is no dimension dispute. All the other operators can be considered as compounded operators with \begin{array}[]{l}MaxMul,\quad(\mbox{matrix product}),\\ Linear,\quad(\mbox{linear mapping}),\\ +,\quad(\mbox{addition, e.g., Feedforward}),\\ Mask,\quad(\mbox{padding-mask, casual-mask}),\\ PE,\quad(\mbox{positional encoding}).\\ \end{array} All these operators require certain dimension-matching conditions. The DFT is assumed to be a transformer with all linear operators, including MatMul, Linear, and + , are all dimension free. A single-layer neural network (NN) is described in Fig. 3 . It can be formulated as \displaystyle\begin{bmatrix}y_{1}\\ \vdots\\ y_{m}\end{bmatrix}=f\left(W\begin{bmatrix}x_{1}\\ \vdots\\ x_{n}\end{bmatrix}+\begin{bmatrix}b_{1}\\ \vdots\\ b_{m}\end{bmatrix}\right), (2) where W\in{\mathcal{M}}_{m\times n} , f(x) : activation function, e.g., f(x)=ReLU(x)=\max(0,x) . [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] [formula] Fig. 3: Single Layer Neural Network From the structure of single-layer NN one sees easily that the key part in an NN consists mainly of MatMul and matrix addition. When the dimensions of matrices or vectors are varying, the DFT needs dimension-varying product and addition. The semi-tensor product (STP) and semi-tensor addition (STA) are the generalizations of conventional matrix product and vector addition, which remove the dimension restriction of conventional matrices and vectors [ 3 ] . Hence STP and STA provide a proper tool to construct DFT. Recently, using STP and STA, a dimension-free matrix theory has been developed [ 3 ] . It has been used to many different fields. For instance, [ 10 , 17 ] introduced its application to Boolean networks; [ 16 ] is a survey on its application to logical networks and finite valued systems; [ 13 ] surveyed its engineering applications; [ 5 ] surveyed its various applications to finite games; [ 21 ] finite automata; and there are some works on cryptography [ 22 ] , compressed sensing [ 20 ] , etc. The matrix-matrix (MM-) STP is a generalization of conventional matrix product, which allows the factor matrices of arbitrary dimensions; the matrix=vector (MV-) STP is a generalization of conventional matrix-vector product, which is also dimension-free; meanwhile, the STA, as a generalization of conventional vector addition, is an addition for vectors of different dimensions. These dimension-free operators from dimension-free matrix theory provide a set of suitable tools to construct DFT. In a transformer, as well as in all AI based signal processes, it is commonly true that the data encountered are of higher order. Hence, describing them as hypermatrices is a proper way. In literature, the hypermatrices are also called the tensors. In this paper, we call them hypermatrices of order k . When the order of a set of data is 1 we have a vector; when the order is 2 , we have a matrix; and as the order k>2 we have k -th order hypermatrices. We refer to [ 14 ] for general hypermatrices, to [ 1 ] for some new applications of hypermatrices. The paper is organized as follows. Section 2 reviews some basic concepts/results from dimension-free matrix theory, which are used in the sequel. They are (i) STP and STA; (ii) Cross-dimensional projection; (iii) Hypermatrices and Hypervectors. (to be completed!) Before ending this section, a list of notations is provided as follows. • {\mathbb{R}} : set of real numbers. • {\mathbb{Z}}_{+} : set of positive integers. • [a,b] , a,b\in{\mathbb{Z}}_{+} , a\leq b : set of integers \{t\in{\mathbb{Z}}_{+}\;|\;a\leq t\leq b . • {\mathcal{M}}_{m\times n} : set of m\times n dimensional real matrices. • \Col_{i}(A) : i -th column of A . • \Row_{i}(A) : i -th row of A . • I_{n} : n\times n Identity matrix. • \delta_{n}^{i} :