Emergent Ordinal Geometry in Transformers Trained on Local Comparisons
Nishit Singh · 2026
Read Emergent Ordinal Geometry in Transformers Trained on Local Comparisons by Nishit Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01269.txt.
Introduction
Singh asks if transformers learn a mental number line from pairs. People infer that A is below C from A below B and B below C. They do so with an inner line, not logic chains. Far pairs feel easier than near pairs. Singh trains small models on adjacent pairs only. A hidden total order sits behind those pairs. He then tests them on far pairs they never saw.
The models generalize to the unseen pairs. Their embeddings collapse onto a one-dimensional curve. Its main axis tracks the hidden rank near perfectly. Confidence and spacing grow with rank distance, like in humans. A large pretrained model shows the same shape. Lines fit sizes and digits, while cycles fit months.
Catalog record
| CREATOR | Nishit Singh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01269
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Emergent Ordinal Geometry in Transformers Trained on Local Comparisons Nishit Singh Abstract Transitive inference is the challenge of inferring that A < C from knowing only adjacent relations (A < B, B < C). It is solved by humans and animals not through logical chaining but via an analogue mental number line, whose signature is the symbolic distance effect: distant comparisons are easier than nearby ones. We ask whether Transformers acquire the same primitive, training small models exclusively on adjacent comparisons from a hidden total order and evaluating generalization to unseen distant pairs. We find that out-of-distribution generalization emerges alongside a striking geometric reorganization: entity embeddings collapse onto a one-dimensional manifold whose principal axis recovers the hidden rank order with near-perfect fidelity, and this structure is sensitive to optimization in ways that produce grokking-like transient dynamics. Critically, even when accuracy is at ceiling, decision confidence and geometric separation both scale monotonically with rank distance, directly mirroring the symbolic distance effect observed across decades of behavioural experiments on humans, primates, and rodents. We further show the same rank-aligned geometry in a pretrained large language model, where it tracks the topology of each ordinal relation: linear for sizes and digits, cyclic for months. These results ground a 50-year-old behavioural regularity in the geometry of learned representations, offering a mechanistic account of transitive inference that bridges cognitive science and modern neural networks. 1 Introduction Knowing that Alice is taller than Bob and Bob taller than Carol, one infers that Alice is taller than Carol without direct comparison. This capacity, transitive inference (TI), is phylogenetically widespread and documented in children ( 2 ), primates ( 8 ), rodents ( 3 ), and birds ( 16 ) and, crucially, is not solved by formal logic. Behavioral work instead supports an analog mental number line: items are arranged along a continuous internal ordering from which comparisons are read off spatially ( 15 ). The signature of this account is the symbolic distance effect: distant comparisons are faster and more accurate than adjacent ones ( 2 ), the opposite of what step-by-step chaining predicts, and exactly what one expects if order is encoded as position on a line. Transformers are under no explicit pressure to represent order this way, yet abstract relational structure is known to emerge in their representations as a byproduct of training ( 10 ; 9 ), sometimes abruptly and late, in grokking-like transitions ( 14 ). We ask a question at the intersection of cognitive science and interpretability: when a Transformer learns to order items from only local comparisons, does it discover the same primitive, a mental number line , and exhibit the same behavioral signatures as biological cognition? Mirroring the original 2 paradigm, in which subjects learn adjacent comparisons and are tested on inferred distant pairs, we train small Transformers only on adjacent relations from a hidden total order and evaluate on held-out distant pairs. We find that out-of-distribution generalization coincides with a geometric reorganization: entity embeddings collapse onto a one-dimensional manifold whose principal axis recovers the hidden rank order. The resemblance is behavioral as well as structural, even at accuracy ceiling, decision confidence and geometric separation scale with rank distance, a computational analogue of the symbolic distance effect. Our contributions are: (i) a minimum-norm account predicting the geometry, including end-anchoring; (ii) A synthetic task, modeled on the classic TI paradigm, isolating ordinal generalization from local comparisons. (iii) Evidence that Transformers spontaneously form a 1D ordinal manifold and reproduce the symbolic distance effect. (iv) evidence that the same geometry appears in a pretrained 1.5B LLM, tracking the topology of each relation. 2 Background Transitive inference and the distance effect: Since 2 , TI has been studied as ordinal rather than logical reasoning; the symbolic distance and end-anchor effects are its canonical behavioral signatures across species ( 16 ). Emergent structure in Transformers: Generalizing solutions can appear abruptly after memorization, a phenomenon termed grokking ( 14 ), driven by weight decay and trackable through internal progress measures as discussed in texts like 11 ; 10 formalize relational reasoning as emergent representational geometry. Linear representations: Semantic and relational structure is often encoded linearly in embedding space ( 13 ; 4 ), motivating our use of principal-component analysis as a probe. 3 Theoretical Framework The rank-ordered one-dimensional manifold is not an accident of this task. It is the configuration that L_{2} regularization must prefer ( 12 ). We make this precise under a standard idealization: the comparison is read from a linear functional of the entity embeddings. We then show that the predicted geometry matches our empirical findings, including a characteristic departure from uniformity that turns out to be a second cognitive signature. Setup Let the relation be decided by a fixed readout direction w\in\mathbb{R}^{d} with \lVert w\rVert=1 , through the scalar score s_{i}\;=\;\langle w,\,h_{e_{i}}\rangle. The model predicts e_{i}\prec e_{j} whenever s_{j}-s_{i}>0 . This score is exactly the quantity our PC1 projection \pi_{i} estimates empirically. Training supervises only adjacent pairs. We encode this as a margin constraint with target margin \gamma>0 : s_{i+1}-s_{i}\;\geq\;\gamma,\qquad 0\leq i<N-1. (1) Weight decay then selects, among all feasible embeddings, the one of minimum norm: \min_{\{h_{e_{i}}\}}\ \sum_{i=0}^{N-1}\lVert h_{e_{i}}\rVert^{2}\qquad\text{subject to \eqref{eq:margin}.} (2) The regularized solution is the rank line Proposition 1 . Every minimizer of ( 2 ) satisfies: (i) Collinearity: h_{e_{i}}=s_{i}\,w for all i ; every off-axis component vanishes. (ii) Monotonicity: s_{0}<s_{1}<\dots<s_{N-1} , so PC1 recovers the hidden order ( r_{\mathrm{rank}}=1 ). (iii) Equal spacing 1 1 1 Under uniform constraint multiplicity. : Every constraint is active, s_{i+1}-s_{i}=\gamma , giving s_{i}=\gamma\,(i-\tfrac{N-1}{2}) . Proof. Step 1 (collinearity): Split each embedding into its component along w and orthogonal to it, h_{e_{i}}\;=\;s_{i}\,w+h_{e_{i}}^{\perp},\qquad\langle w,\,h_{e_{i}}^{\perp}\rangle=0, so that \lVert h_{e_{i}}\rVert^{2}=s_{i}^{2}+\lVert h_{e_{i}}^{\perp}\rVert^{2} . The constraints ( 1 ) depend only on the scores s_{i} , never on the orthogonal parts. Setting every h_{e_{i}}^{\perp}=0 therefore preserves feasibility while strictly lowering the objective unless it is already zero. Hence h_{e_{i}}=s_{i}w at any optimum. The problem collapses to a convex quadratic program in the scalars alone, \min_{\{s_{i}\}}\ \sum_{i}s_{i}^{2}\qquad\text{s.t.}\qquad s_{i+1}-s_{i}\geq\gamma, which has a unique minimizer. Step 2 (equal spacing): Suppose for contradiction that some gap is slack: s_{k+1}-s_{k}>\gamma for some 0\leq k<N-1 . Consider shifting the upper block \{s_{k+1},\ldots,s_{N-1}\} down by \epsilon>0 and the lower block \{s_{0},\ldots,s_{k}\} up by \delta>0 , where (N-1-k)\,\epsilon=(k+1)\,\delta, so the total shift is zero. Every gap within each block is unchanged, the slack gap at k widens, and all other tight gaps remain at \gamma , so the move is feasible. The change in the objective is \Delta=-2\epsilon\sum_{m>k}s_{m}+2\delta\sum_{m\leq k}s_{m}+O(\epsilon^{2}). Substituting \delta=\frac{(N-1-k)}{(k+1)}\epsilon and taking \epsilon\to 0^{+} , the sign of \Delta is determined by (k+1)\sum_{m>k}s_{m}-(N-1-k)\sum_{m\leq k}s_{m}. If this quantity is positive, the original shift decreases the objective; if negative, reversing the shift (moving the upper block up and lower block down) decreases it. It is zero only when \frac{1}{N-1-k}\sum_{m>k}s_{m}=\frac{1}{k+1}\sum_{m\leq k}s_{m}, i.e. the two block means are equal, which combined with s_{k+1}>s_{k}+\gamma>s_{k} is impossible for \gamma>0 . In either case a feasible descent direction exists, contradicting optimality. Every gap is therefore tight: s_{i+1}-s_{i}=\gamma for all i . Step 3 (monotonicity and centering): Equal gaps give s_{i}=s_{0}+\gamma i , which is increasing in i . Minimizing \sum_{i}(s_{0}+\gamma i)^{2} over the offset s_{0} yields s_{i}=\gamma\,(i-\tfrac{N-1}{2}) . ∎ What the proposition explains