Weber’s Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models
Jon-Paul Cacioli · 2026
Read Weber’s Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models by Jon-Paul Cacioli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20642.txt.
Introduction
Cacioli studies how transformer language models store magnitude. Past work clashes: some find log spacing, some linear, some digit based rings. This study applies psychophysics tools across three models and three domains. Log style geometry won in all 96 test cells. Yet behavior did not match geometry in every model. One model showed a human like Weber fraction, the other did not. Early layers drove magnitude work, while later layers held the strongest geometry.
Catalog record
| CREATOR | Jon-Paul Cacioli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57367 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20642
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Weber’s Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models Jon-Paul Cacioli Abstract How do transformer language models represent magnitude? Recent work disagrees: some find logarithmic spacing, others linear encoding, others per-digit circular representations. We argue that the disagreement is methodological and apply the formal tools of psychophysics to resolve it. Using four converging paradigms (representational similarity analysis, behavioural discrimination, precision gradients, causal intervention) across three magnitude domains (numerical, temporal, spatial) in three 7–9B instruction-tuned models spanning three architecture families (Llama, Mistral, Qwen), we report three findings. First, representational distance structure is consistently log-compressive: RSA correlations with a Weber-law dissimilarity matrix ranged from .68 to .96 across all 96 model–domain–layer cells (2 models × \times 3 domains × \times 16 primary layers), with both cosine and Euclidean metrics agreeing and linear geometry never preferred. Second, this geometry is dissociated from behaviour: one model produces a human-range Weber fraction (WF = 0.20, i.e. ∼ \sim 20% relative difference threshold) while the other does not, and both models perform at chance on temporal and spatial discrimination despite possessing logarithmic geometry for those domains. Third, causal intervention reveals a layer dissociation: early layers where geometry is weakest are functionally implicated in approximate magnitude processing (4.1 × \times specificity), while later layers where geometry is strongest are not causally engaged under our linear intervention (1.2 × \times ). Corpus analysis confirms the efficient coding precondition: integer frequencies in the training data follow a power law ( α \alpha = 0.77). These results are consistent with the hypothesis that training data statistics (without biological resource constraints) are sufficient to produce log-compressive magnitude geometry, but that geometry alone does not guarantee behavioural competence. 1 Introduction Transformer language models process numerical, temporal, and spatial magnitudes as part of ordinary language understanding. Recent work has begun to characterise how these magnitudes are internally represented, but the emerging picture is contradictory. 1 extracted principal components from hidden states and found logarithmic spacing. 20 trained linear probes and concluded the encoding is linear. 10 demonstrated per-digit circular representations in base 10, suggesting that apparent magnitude geometry is an artefact of digit-level structure. 7 found convergent sinusoidal representations across model families, pointing to a Fourier decomposition. 18 showed that number representations entangle Levenshtein string edit distance with numerical distance. These findings are not necessarily contradictory. They may reflect different methodological windows onto the same underlying representation. But the literature lacks a principled framework for characterising the functional form of magnitude representations and their relationship to behaviour. We propose that psychophysics provides this framework. Psychophysics is the quantitative study of the relationship between stimulus magnitudes and their mental representations. Its central law, Weber’s Law ( 19 ), states that the just-noticeable difference between two stimuli is proportional to their magnitude: \Delta S / S = k . 12 first demonstrated that this ratio-dependent regularity extends to symbolic number comparison in humans: reaction times for judging which of two digits is larger decrease with their numerical distance, consistent with an analogue magnitude representation obeying Weber’s Law. In modern cognitive neuroscience, this regularity is understood as a consequence of efficient coding : when a system with limited capacity encodes a variable whose natural distribution follows an approximate 1/ s power law, the information-maximising encoding is logarithmic, and logarithmic encoding produces Weber-like discrimination ( 5 ; 9 ). The efficient coding framework makes a precise, testable prediction: if the input distribution is scale-invariant and the system optimises an information-theoretic objective, the encoding should be logarithmic. Key psychophysical concepts for ML readers. A Weber fraction (WF) is the ratio \Delta S/S at which a subject can discriminate two magnitudes at threshold (e.g., WF = 0.20 means a \sim 20% relative difference is needed). Stevens’ exponent \beta characterises the power-law compression of an internal representation: \beta = 1 is linear, \beta \to 0 is logarithmic, and 0 < \beta < 1 is intermediate compression. Representational similarity analysis (RSA; 8 ) compares a model’s pairwise distance matrix (the representational dissimilarity matrix, or RDM) to theoretical prediction matrices using rank correlation. A Mantel test assesses RSA significance via permutation, accounting for the non-independence of pairwise distances that share endpoints, which is critical when the RDM has C( n , 2) entries but only n independent observations. Transformers meet the distributional precondition. Numerical magnitudes in natural text follow approximately Benford’s Law, with small numbers vastly more frequent than large ones ( 3 ). But transformers lack the biological constraint: there is no metabolic cost, no firing-rate bound, no fixed population size. This creates a natural experiment. If data statistics alone are sufficient to produce logarithmic representations under gradient-based optimisation, the models should develop Weber-like magnitude geometry. If biological constraints are necessary, they should not. We frame this as a test of functional convergence: do different optimisation regimes, operating on input with the same distributional structure, converge on the same representational geometry? Critically, we do not merely ask whether representations are “logarithmic or linear”, a question that existing methods can answer. We ask a richer set of questions that psychophysics was designed for: Does the geometry translate into ratio-dependent behaviour? Does the system allocate more representational precision to small magnitudes? Is the magnitude subspace causally relevant to comparison? Does the encoding generalise across magnitude domains? These questions require the methodological toolkit of psychophysics: Weber fractions, psychometric functions, representational similarity analysis, precision gradients, and causal intervention. 1.1 Contributions Our contributions are as follows: 1. A psychophysics-grounded framework for probing magnitude representations. We apply Weber fractions, psychometric functions, RSA with Mantel permutation tests, precision gradients, and causal intervention to LLM hidden states. To our knowledge, this is the first application of the full psychophysical toolkit to this problem, with pre-registered hypotheses and all results reported regardless of outcome. 2. Empirical resolution of conflicting findings. Logarithmic geometry is consistent across all 96 model–domain–layer cells tested (both distance metrics agreeing), with linear geometry never preferred, resolving the apparent 1 – 20 contradiction. Corpus analysis confirms the efficient coding precondition ( \alpha = 0.77). Variance partitioning quantifies the digit-encoding confound ( 10 ) as real but subordinate to magnitude compression. 3. Three informative dissociations. (a) Geometry without behavioural competence: both models possess logarithmic temporal and spatial geometry but perform at chance on discrimination tasks for those domains. (b) Model-specific readout: Llama produces a human-range Weber fraction; Mistral does not. (c) Causal layer inversion: early layers are functionally implicated in approximate comparison; later layers where geometry is strongest are not causally engaged under our patching intervention. 2 Related Work 2.1 Number Representations in Language Models Representation geometry. 1 used PCA and partial least squares to show that LLM hidden states embed numbers on a logarithmically compressed manifold. 20 reached the opposite conclusion using linear probes, finding that probe accuracy was best predicted by linear, not logarithmic, target functions. We show that both findings are compatible: a smooth logarithmic encoding is globally logarithmic ( 1 is right about the geometry) while being locally linear enough for linear probes to succeed ( 20 is right that probes recover values). The key distinction is methodological: global geometry measures (RSA, model comparison) versus local recovery measures (probe accuracy). For a comprehensive taxonomy of number encoding