Direction-Preserving Number Representations
Bardia Zadeh 1 , George A. Constantinides 1 · 2026
Read Direction-Preserving Number Representations by Bardia Zadeh 1 , George A. Constantinides 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07662.txt.
Introduction
Bardia Zadeh and George A. Constantinides ask a geometric question about low-precision numbers, the tiny formats that make modern machine learning fast. When each component of a vector is drawn from a small shared alphabet of values, how well can the vector's direction be represented? Direction accuracy governs the accuracy of the vector operations that AI hardware performs constantly.
Using a new geometric framework, they quantify how far such product-structured codes fall short of ideal spherical codes, in both low and high dimensions. Surprisingly, standard formats like two's complement, fixed-point, and floating-point are provably suboptimal within their class. Numerically optimized alphabets do better, though NVIDIA's four-bit E2M1 choice comes close to optimal, which explains its strong practical performance. Formal proofs are provided in Lean.
Catalog record
| CREATOR | Bardia Zadeh 1 , George A. Constantinides 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07662
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Direction-Preserving Number Representations Bardia Zadeh 1 , George A. Constantinides 1 Abstract Low-precision number formats are widely used in modern machine learning systems due to their efficiency. Accurate direction representation is key to the accuracy of vector operations. This work precisely explores the extent to which the direction of a vector can be represented by selecting its scalar elements from a common finite alphabet of a given size. This is standard practice in machine learning, where low-precision significands may be narrow-width floating-point or integer values. A geometric framework is introduced for analyzing the directional coverage of such product-structured codes. This work analytically quantifies the suboptimality gap between such product-structured codes and spherical codes for the vector as a whole, in both low and asymptotically high dimensions. Furthermore, within the product code class, it is proven that the standard formats of two’s complement, fixed-point, and floating-point are suboptimal, again with quantified gap, pointing to the potential to develop new scalar number formats. Such scalar alphabets are numerically optimized across multiple block dimensions for directional coverage, including the dimension used in NVIDIA’s NVFP4 format. Experimental results are presented comparing the performance of standard formats and the optimized alphabet. We find that for four bits, NVIDIA’s choice of E2M1 closely approximates the optimized alphabet, providing a geometric explanation for its strong performance in low-precision machine learning workloads and an analytical understanding of the link between that superiority and block size. We provide open-source formal proofs in Lean for the theorems in this work, along with the experimental code and the optimized alphabets obtained. I Introduction Fig. 1: The directional coverage for a product code formed from two elements in the 4-bit E2M1 format. Each intersection of the grid lines corresponds to a representable direction through the product structure, projected onto the circle as a red dot. It can be seen that the red dots are not equally spaced in angle, leading to gaps in angular coverage. The increasing computational demands of modern machine learning systems have driven widespread adoption of low-precision numerical representations during both inference and training. Low-precision representations enable significant savings in power consumption and resource utilization along with increases in throughput, often with minimal degradation in model task accuracy [ 18 ] , [ 2 ] , [ 6 ] . To enable low-precision representation, per-element scalar quantization is commonly used in which real-valued vectors are mapped elementwise to a scalar alphabet, resulting in a product code of quantized elements. When evaluating these quantized vectors, most work focuses on the mean-squared error or other norm-based metrics [ 18 ] . However, the new industrial dominance of block-based number representations such as block minifloat [ 4 ] , MX [ 15 ] , and the draft IEEE P3109 standard [ 10 ] , where scaling of a (sub)vector is recorded separately from the vector itself, suggests an alternative metric: directional fidelity. The product structure of the quantized vectors fundamentally restricts the geometry of the representation space: the set of achievable directions is not arbitrary, but is instead determined by the scalar alphabet. Figure 1 shows the coverage of a product code formed by two 4-bit E2M1 1 1 1 E2M1 refers to a floating-point format with exponent width 2 and mantissa width 1. This format is used in NVFP4 [ 12 ] . E2M1 does not support infinities or NaN values, and is part of the OCP standard [ 15 ] . elements. These observations motivate a shift in perspective for the design of number systems themselves: rather than focusing on traditional questions of dynamic range and precision, we ask whether scalar representations can be designed to preserve vector directions when those scalars populate the elements of a vector . We offer the following contributions in this work: 1. An analytical proof of the angular error gap, in both low and high dimensions, between such product-structured codes and optimal spherical codes. 2. An analytical proof that among all product codes for a given bit-width, two’s complement, fixed-point, and floating-point are suboptimal, with quantified suboptimality gap. 3. A numerical optimization approach to obtain a direction-preserving alphabet, and an empirical evaluation of contemporary 4-bit formats against this benchmark. 4. Open-source access to the following: proofs of our theorems in Lean, experimental code for evaluating various formats, and the optimized number formats themselves 2 2 2 https://github.com/bardia01/Direction-Preserving-Number-Representations . II Background and Related Work II-A Scalar Quantization of Vectors Scalar quantization applied to a vector independently maps each element to a discrete set of values. Formally, given a vector x\in\mathbb{R}^{d} , we apply a scalar quantizer Q(\cdot) element-wise, producing a quantized vector \hat{x} such that \hat{x}_{i}=Q(x_{i}) for each dimension i . The quantizer Q(\cdot) can be uniform, or non-uniform, symmetric or asymmetric [ 6 ] . Within the scope of this work, a scalar quantizer maps an element to its nearest neighbor in the alphabet: Q:\mathbb{R}\to\mathcal{S} , where \mathcal{S}\subset\mathbb{R} denotes the set of representable values. We assume the existence of a tie-break rule, but the nature of that rule is irrelevant to this work. Thus we may write: Q(x)=\arg\min_{s\in\mathcal{S}}|x-s|. The resulting quantized vector is drawn from the Cartesian product \mathcal{S}^{n} ; in this sense, it forms a product code. The set of values a scalar quantizer can map to is referred to as the scalar alphabet. II-B Spherical codes Spherical codes [ 17 ] are sets of points distributed on the surface of a unit sphere. In this work, spherical codes serve as an unconstrained benchmark. For a fixed number of representable directions, a spherical code may place points arbitrarily on the sphere and therefore captures the best achievable directional coverage. In contrast, the product code direction sets considered in this paper are constrained in their angular coverage capabilities due to their construction by per-coordinate scalar choices drawn from a shared finite alphabet. II-C Shape–gain quantization and block-formats The decomposition of a vector into its magnitude (gain) and direction (shape) is a classical approach in vector quantization [ 7 ] . Shape–gain quantizers encode the two components separately, typically using scalar quantization for the gain and spherical codes for the shape component. By factoring out a common scale across groups of elements, block-based number representations [ 4 ] , [ 15 ] , [ 10 ] may be viewed as instances of shape–gain quantization in which the shape component is constrained to a product-structured code. In this paper we investigate the implications of using product structure to represent the shape component, as opposed to the standard practice of using spherical codes. II-D Direction Preservation in Low-Precision Representations Recent work has increasingly highlighted the importance of preserving directional information in low-precision machine learning workloads. Several works explicitly incorporate direction or inner-product preservation into the quantization objective. TurboQuant [ 19 ] includes a secondary optimization stage that directly minimizes inner-product error between original and quantized vectors. PolarQuant [ 8 ] parametrizes vectors in polar coordinates and quantizes magnitude and direction separately to better preserve angular structure, similarly to traditional shape-gain quantizers. These approaches focus on designing quantization schemes that preserve directional properties. Instead, we analytically characterize and study the representational limits imposed by a fixed class of quantizers, namely those derived from a product structure of scalar