A short survey on almost orthogonal vectors in a few specific large dimensions

ℝ \mathbb{R} ami Luisto · 2025

Read A short survey on almost orthogonal vectors in a few specific large dimensions by ℝ \mathbb{R} ami Luisto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23609.txt.

Introduction

Rami Luisto surveys almost orthogonal vectors, directions in space whose cosine similarity sits near zero, meaning they point nearly at right angles to each other. The idea links pure mathematics with coding theory, where it appears as spherical packing and spherical codes. Fresh interest comes from AI: large language models seem to store distinct concepts as nearly orthogonal directions in their high-dimensional embedding spaces.

The survey views the topic from three angles. First comes the mathematical theory of almost orthogonality itself, with its bounds and constructions. Second, Luisto reports observations from the embedding spaces of real language models, where the near-orthogonal structure shows up in practice. Third, he describes simulation methods that generate large sets of such vectors. By keeping the treatment to a few chosen dimensions, the survey stays concrete while connecting three communities that rarely meet.

Catalog record

CREATORℝ \mathbb{R} ami Luisto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A short survey on almost orthogonal vectors in a few specific large dimensions ℝ \mathbb{R} ami Luisto Abstract . The concept of almost orthogonal vectors , i.e. vectors whose cosine similarity is close to 0 0 , relates to topics both in pure mathematics and in coding theory under the guises of spherical packing and spherical codes. In recent years the rise of advanced language models in AI has created new interest in this concept as the models seem to store certain concepts as almost orthogonal directions in high-dimensional spaces. In this survey we represent some ideas regarding almost orthogonal vectors through three approaches: (1) the mathematical theory of almost orthogonality, (2) some observations from the embedding spaces of language models, and (3) generation of large sets of almost orthogonal vectors by simulations. 2010 Mathematics Subject Classification 52C17 (68T07, 60D05, 46B85, 94B65) 1. Introduction In a given Euclidean space \mathbb{R}^{n} there can exist at most n directions that are pairwise orthogonal [ Axl15 ] . If we relax the condition of orthogonality by requiring the inner product of unit vectors to be merely close to 0 the situation changes drastically, especially in higher dimensions. This phase transition in high and low dimensions relates to a concept sometimes known as the curse (or blessing) of high dimensionality . We’ll discuss this further in the coming sections, see especially Section 3.3 We say that two vectors \mathbf{v},\mathbf{w}\in\mathbb{R}^{n} are \varepsilon -almost orthogonal for \varepsilon>0 if their cosine similarity lies between -\varepsilon and \varepsilon . The main motivating question for this paper is as follows: How many pairwise \varepsilon -almost orthogonal vectors can fit in \mathbb{R}^{n} for various values of \varepsilon and n ? As we’ll discuss in Section 4.3 this question is closely tied to various open problems related to both spherical codes and sphere packing . In particular, exact solutions to the best configurations of vectors are beyond the scope of this survey, and instead we will focus on various estimates. Since e.g. spherical codes have many practical applications, there already exists literature on approximate solutions. However, these do tend to focus more on low-dimensional examples – see e.g. [ Coh23 ] where the examples only go up to dimension 32. This is natural if you aim to use this with e.g. binary blocks that tend to have word sizes of 8, 16, or 32 bits. By contrast, the existing results for almost orthogonal vectors in higher dimensions tend to focus on the asymptotic behavior of the results, i.e. what happens when the ambient dimension grows very large. In particular, the typical results like the Johnson-Lindenstrauss Lemma tend not to yield good estimates in dimensions below several thousands – see Section 4.2 . The contextual motivation for us, on the other hand, arises from the so called embedding vectors of various AI-models. For a contemporary example, various language models based on the transformer architecture (including GPT, LLaMa, Gemini and Mistral) encode the meaning of text through embedding vectors that live in high-dimensional Euclidean spaces like \mathbb{R}^{768} or \mathbb{R}^{1024} (see e.g. [ Eth19 , Ope23 , RKR21 ] ). These dimensions lie between the small dimensions of coding theory and high dimensions of asymptotical results. This gap is of primary interest in this survey. Based on current understanding, see e.g. [ EHO+22 , HCH+23 , BTB+23 , TCM+24 ] , modern language models models seem to store different semantic concepts as different directions in these embedding spaces. Crucially for us, these different concepts seem to be encoded as almost orthogonal vectors, and it turns out that the geometry of e.g. \mathbb{R}^{768} can encode much more such directions than just 768, though lossily. A driving motivation in this survey is to better understand how many almost orthogonal vectors can be fitted in various high-dimensional Euclidean spaces where the particular dimensions we study are the embedding space dimensions of particular contemporary language models – for the sake of exposition we will often focus on dimensions 512, 768, 1024 and 2048 when concrete examples are needed. As per the parameter \varepsilon in \varepsilon -almost orthogonality we will in most examples focus on either \varepsilon=0.1 or \varepsilon=0.01 when we need to fix the parameter ourselves. This is again purely for the sake of exposition, as the values themselves do not carry any special meaning. Note that \varepsilon=0.1 corresponds to angles between 84 and 96 degrees, while \varepsilon=0.01 corresponds to angles of roughly 90\pm 0.5 degrees. Also note that if unit vectors \mathbf{v} and \mathbf{w} have an inner product of 0.1 , then \mathbf{v}=0.1\mathbf{w}+0.9\mathbf{u} where \mathbf{u} is orthogonal to \mathbf{w} , meaning that ”10% of \mathbf{v} is explained by \mathbf{w} ”. Part of the codebase used for the simulations in this work is available at https://github.com/ramiluisto/AlmostOrthogonalGenerators . 1.1. Structure of this survey We’ll start by ”setting the scene” and describe in a bit more detail on why we are interested in almost orthogonality in high-dimensional spaces when studying modern (Large) Language models. We will then go through some mathematical preliminaries, including some more heuristical comments on how the geometry of low and high dimensions differs. We’ll then move on to studying some (trivial) mathematical bounds we can get to the amount of almost orthogonal vectors in Euclidean spaces. We then look at some other classical results, in particular the Johnson-Lindenstrauss Lemma. The bounds we see here turn out to be somewhat meagre compared to practical situations. Thus we turn into simulations and compare various methods of generating almost orthogonal vectors. In the final section we’ll draw conclusions on all of this. We wish to emphasize that this paper is structured as a survey ; it claims no new mathematical theorems, rather it tries to provide a convenient summary of useful known information. On the other hand it does not claim to be a comprehensive survey as many areas are only briefly mentioned, and we lack the expertise to comment authoratively to many topics discussed here. Acknowledgements: This work started as an appendix to a master’s thesis. We would like to thank the advisors Sami Äyramö and Ida Toivanen for their support. We furthermore extend our thanks to Ilkka Pölönen for discussions on the topic. 2. Context – Latent spaces of (Large) Language Models We will not repeat here a detailed introduction to the architecture of transformer models, rather we given a quick overview of the terminology. For a proper introduction, see e.g. [ TVWW22 ] . Most classical transformer-based language models break incoming text into tokens . These tokens tend to be common words or word fragments. The collection of tokens a model has is called the vocabulary . The vocabulary size of e.g. the different BERT 1 1 1 BERT is what I would call the archetypical small language model, see e.g. [ RKR21 ] for a comprehensive text on the topic. variants are in the ballpark of 30–50 thousand tokens. For a more thorough introduction to tokens and their usage we refer to [ DCLT18 ] and [ Mik13 ] . For each token in the vocabulary the model has a learned embedding vector of a particular dimension d_{\text{model}} . These vectors are called the input embeddings of the model. Thus a given input sequence of

Read the complete text