When Models Manipulate Manifolds: The Geometry of a Counting Task Wes Gurnee , Emmanuel Ameisen , Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, Joshua Batson[1em] Anthropic, Equal contribution. Corresponding author: joshb@anthropic.com Abstract Language models can perceive visual properties of text despite receiving only sequences of tokens—we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads ‘‘twist” these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover ‘‘visual illusions”—character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability. 1 1 1 This is an archival version of a paper originally published on October 21, 2025 at Transformer Circuits , which we recommend for its improved interactivity and styling. 1 Introduction Intelligent systems need perception to understand, predict, and navigate their environment. These sensory capabilities reflect what’s useful for survival in a specific environment: bats use echolocation, migratory birds sense magnetic fields, Arctic reindeer shift their UV vision seasonally. But when your world is made of text, what do you see? Language models encounter many text-based tasks that benefit from visual or spatial reasoning: parsing ASCII art, interpreting tables, or handling text wrapping constraints. Yet their only “sensory” input is a sequence of integers representing tokens. They must learn perceptual abilities from scratch, developing specialized mechanisms in the process. In this work, we investigate the mechanisms that enable Claude 3.5 Haiku to perform a natural perceptual task which is common in pretraining corpora and involves tracking position in a document. We find learned representations of position that are in some ways quite similar to the biological neurons found in mammals who perform analogous tasks (“place cells” and “boundary cells” in mice), but in other ways unique to the constraints of the residual stream in language models. We study these representations and find dual interpretations: we can understand them as a family of discrete features or as a one-dimensional “feature manifold”/“multidimensional feature” [ 57 , 58 , 21 , 17 ] . 2 2 2 All features have a magnitude dimension; so a discrete feature is a one-dimensional ray, and a one-dimensional feature manifold is the set of all scalings of that manifold, contracting to the origin. See What is a Linear Representation? What is a Multidimensional Feature? In the first interpretation, position is determined by which features activate and how strongly; in the latter interpretation, it’s determined by angular movement on the feature manifold. Similarly, computation has two dual interpretations, as discrete circuits or geometric transformations. Figure 1: Left: a description of the linebreaking task. Center: example feature families we study. Right: a highlight of the geometry of representations and computations. The task we study is linebreaking in fixed-width text. When training on source code, chat logs, email archives, scanned articles, or judicial rulings that have line width constraints, how does the model learn to predict when to break a line? 3 3 3 Michaud et al. looked for “quanta” of model skills by clustering gradients [ 46 ] . Their Figure 1 shows that predicting newlines in fixed-width text formed one of the top 400 clusters for the smallest model in the 70m Pythia model. Human visual perception lets us do this almost completely subconsciously – when writing a birthday card, you can see when you are out of room on a line and need to begin the next – but language models see just a list of integers. In order to correctly predict the next token , in addition to selecting the next word , the model must somehow count the characters in the current line, subtract that from the line width constraint of the document, and compare the number of characters remaining to the length of the next word. As a concrete example, consider the below pair of prompts with an implicit 50-character line wrapping constraint. 4 4 4 The wrapping constraint is implicit. Each newline gives a lower bound (the previous word did fit) and an upper bound (the next word did not). We do not nail down the extent to which the model performs optimal inference with respect to those constraints, rather focusing on how it approximately uses the length of each preceding line to determine whether to break the next. There are also many edge cases for handling tokenization and punctuation. A model could even attempt to infer whether the source document used a non-monospace font and then use the pixel count rather than the character count as a predictive signal! When the next word fits, the model says it; when it does not, the model breaks the line: Figure 2: A pair of prompts highlighting the difficulty of the linebreaking task. To orient ourselves to the stages of the computation, we first studied the model using discrete dictionary features. In this frame, we can understand computation as an “attribution graph” [ 2 ] where a cascade of features excite or inhibit each other. 5 5 5 We actually first tried to use patching and probing without looking at the graph as a kind of methodological test of the utility of features, but did not make much progress. In hindsight, we were training probes for quantities different than the ones the model represents cleanly, e.g., a fusion of the current token position and the line width. Figure 3: Attribution Graph for Claude 3.5 Haiku’s prediction of a newline in the aluminum prompt. We see features relating to “width of the previous line” and “position in the current line” which together activate features for “distance from line limit”. Combined with features for the planned next word, these features activate “predict newline” features. The attribution graph shows how the model performs this task by combining features that represent different concepts it needs to track: 1. Features for the current position in the line (the character count) as well as features for the total line width (the constraint) are computed by accumulating features for individual token lengths. 2. The model then combines these two representations — current position and line width — to estimate the distance from the end of the line, leading to “characters remaining” features. 3. Finally, the model uses this estimate of characters remaining along with features for the planned next word to determine if the next word will fit on the line or not. The attribution graph provides a kind of execution trace of the algorithm, showing on this prompt which variables are computed and from what. After finding large feature families involved in representing these quantities across a diverse dataset, we suspected a simpler lens might be provided in terms of lower-dimensional feature manifolds interacting geometrically. We found geometric perspectives on the following questions: Figure 4: Key steps in the linebreaking behavior can be described in terms of the construction and manipulation of manifolds. How does the model represent different counts? The number of characters in a token, the number of characters in the current line, the overall line width constraint, and the number of characters remaining in the current line are each represented on 1-dimensional feature manifolds embedded with high curvature in low-dimensional subspaces of the residual stream. These manifolds have a dual interpretation in terms of discrete features, which tile the manifold in a canonical way, providing approximate local coordinates. Manifolds with similar geometry arise for a variety of ordinal concepts, and a ringing pattern we see in the embedded geometry in all these cases is optimal with respect to a simple physical model (§ Representing Character Count ). 6 6 6 Ringing, in the manifold perspective, corresponds to interference in the feature superposition perspective. How does the model detect the boundary? To detect an approaching line boundary, the model must compare two quantities: the current character count and the line width. We find attention heads whose QK matrix rotates one counting manifold to align it with the other at a specific offset, creating a large inner product when the difference of the counts falls within a target range. Multiple heads with different offsets work together to precisely estimate the characters remaining (§ Sensing the Line Boundary ). How does the model know if the next word fits? The final decision — whether to predict a newline — requires combining the estimate of characters remaining with the length of the predicted next word. We discover that the model positions these counts on near-orthogonal subspaces, creating a geometric structure where the correct linebreak prediction is linearly separable (§ Predicting the Newline ). How does the model construct these curved geometries? The curvature in the character count representation manifold is produced by many attention heads working together, each contributing a piece of the overall curvature. This distributed algorithm is necessary because individual components cannot generate sufficient output variance to create the full representation (§ A Distributed Character Counting Algorithm ). We validate these interpretations through targeted interventions, ablations, and “visual illusions” — character sequences that hijack specific attention mechanisms to disrupt spatial perception (§ Visual Illusions ). Zooming out, we take several broader lessons from this mechanistic case study: When Models Manipulate Manifolds . For representing a scalar quantity (e.g., integer counts from 1 to N ), it is inefficient to use N orthogonal dimensions, and not expressive enough to use just one 7 7 7 Orthogonal dimensions would also not be robust to estimation noise. . Instead models learn to represent these quantities on a feature manifold with intrinsic dimension 1 (the count) embedded in a subspace with extrinsic dimension 1> }} ;| || ’, @@ . We further analyzed the extent to which there was a relationship between ‘distraction’ of the important attention heads and the impact on the newline prediction. Indeed we found that many of the sequences with potent modulation of newline probability—and especially code-related character pairs—also exhibited substantial modulation of attention patterns. Figure 33: Insertion of most pairs of characters only moderately impacts the probability of predicting a newline. A subset of pairs, most of which appear related to code or delimiters, substantially disrupt newline prediction. The impact on newline prediction (originally 0.79) is correlated with how much inserted tokens ‘distracts’ character counting attention heads. While in the aluminum prompt the task is implicit, this illusion generalizes to settings where the comparison task is made explicit. These direct comparisons are perhaps more analogous to the Ponzo, Sander, and Müller-Lyer illusions, where the perception and comparison is more direct. Figure 34: Prediction for explicit length comparison task with normal prompt (left) and with visual illusion insert (right). These effects are robust to multiple choice orderings. Moreover, if the length of the text following the @@ exceeds that of the alternative choice, the alternative choice is selected as being shorter. While we are not claiming any direct analogy between illusions of human visual perception and this alteration of line character count estimates, the parallels are suggestive. In both cases we can see the broader phenomena of contextual cues, and the application of learned priors about those cues, modulating estimates of object properties of entities. In the human case, priors such as three-dimensional perspective can influence perception of object size, or color constancy can influence estimates of luminance (such as in the checker shadow illusion). Here, one possible interpretation of our results is that mis-application of a learned prior, including the role of cues such as @@ in git diffs, can also modulate estimates of properties such as line length. 7 Related Work Objective . This work is at the intersection of LLM “biology” (making empirical observations about what is going on inside models; e.g. [ 43 , 63 ] ) and low level reverse engineering of neural networks (attempting to fully characterize an algorithm or mechanism; e.g. [ 54 , 72 , 52 , 41 ] ). Methodologically, our work makes heavy use of attribution graphs [ 2 , 20 , 14 ] with QK attributions [ 36 ] built on top of crosscoders [ 44 ] . Linebreaking . Michaud et al. [ 46 ] identified linebreaking in fixed-width text as one of the top 400 “quanta” of model behavior in the smallest model (70m parameters) in the Pythia suite. Position. Prior interpretability work on positional mechanism has largely focused on token position (e.g., [ 75 , 76 , 71 , 10 , 25 ] ). These works have shown that there exist MLP neurons [ 71 , 25 ] , SAE features [ 10 ] , and learned position embeddings [ 75 ] with periodic structure encoding absolute token position. Our work illustrates how a model might also want to construct non-token based position schemes that are more natural for many downstream prediction tasks. Others have also studied, even going back to LSTMs, the existence of mechanisms in language models for controlling the length of output responses [ 66 , 49 ] , as well as performed more theoretical analyses of the space of counting algorithms [ 69 , 8 ] . Geometry and Feature Manifolds. Beyond position, there has been extensive work in understanding the geometric representation of numbers, especially in toy models (e.g., [ 52 , 78 , 50 ] ) and in the context of arithmetic in LLMs (e.g., [ 68 , 79 , 53 , 37 , 34 ] ). Collectively, these works have shown that both real LLMs and toy transformers learn periodic representations [ 79 , 37 , 34 ] , with numbers arranged in a helix to enable certain matrix multiplication based addition algorithms [ 52 , 37 ] , and that these representations are provably optimal in certain settings [ 50 ] . In our context, we similarly observe helical representations [ 37 ] , numeric dilation [ 1 ] , and distributed algorithms across components that collectively implement a correct computation [ 28 , 34 ] . Multidimensional features with clear geometric structure have been found in more natural contexts [ 23 , 17 , 48 ] , like in the representation and computation of certain ordinal relationships (e.g., months of the year). In vision models, curve detector neurons [ 5 ] and features [ 22 ] have been especially well studied and closely resemble the kind of discretization we observe with the families of character counting features. Many other topics have received interpretability analysis of the underlying geometry, such as grammatical relations [ 31 , 11 ] , multilingual representations [ 7 ] , truth [ 45 ] , binding [ 18 ] , refusal [ 74 ] , features [ 32 , 42 ] , and hierarchy [ 59 ] , though more conceptual research is needed [ 73 ] . Perhaps most relevant is recent work from Modell et al. [ 48 ] , who provide a more formal notion of a feature manifold, and propose that cosine similarity encodes the intrinsic geometry of features. When testing their theory, they observe highly structured and interpretable data manifolds that have ripples and dilation, similar to our counting manifolds. These observations raise a methodological challenge in how to best capture data with different structure (see e.g. [ 32 , 47 , 35 ] ), but also the exciting hypothesis that many naturally continuous variables (e.g., [ 30 , 27 ] ) exist in more organized manifolds. Biological Analogues. The geometric and algorithmic patterns we observe have suggestive parallels to perception in biological neural systems. Our character count features are analogous to place cells on a 1-D track [ 51 ] and our boundary detecting features are analogous to boundary cells [ 67 ] . These features exhibit dilation—representing increasingly large character counts activating over increasingly large ranges—mirroring the dilation of number representations in biological brains [ 13 , 62 ] . Moreover, the organization of the features on a low dimensional manifold is an instance of a common motif in biological cognition (e.g., [ 61 ] ). While the analogies are not perfect, we suspect that there is still fruitful conceptual overlap from increased collaboration between neuroscience and interpretability [ 70 , 29 , 40 ] . 8 Discussion In this paper, we studied the steps involved in a large model performing a naturalistic behavior. The linebreaking task, frequently encountered in training, requires the model to represent and compute a number of scalar quantities involving position in character count units that are not explicit in its input or output 28 28 28 Tokens do not come annotated with character counts, and there are no vertical bars on the page showing the line width. , then integrate those values with the outputs of complex semantic circuits (that predict the next proper word) to predict the next token. We found sparse features corresponding to each important step of the computation, and for those steps involving scalar quantities, we were able to find a geometric description that significantly simplified the interpretation of the algorithm used by the model. We now reflect on what we learned from that process: Naturalistic Behavior and Sensory Processing . Deep mechanistic case studies benefit from choosing behaviors that the model performs consistently well, as these are more likely to have crisper mechanisms. This means prioritizing tasks that are natural in pretraining over tasks that seem natural to human investigators, and ideally, that are easily supervisable. As in biological neuroscience, perceptual tasks are often both natural and easy to supervise for interpretability (e.g., it is easy to modify the input in a programmatic way). Although we sometimes describe the early layers of language models as responsible for “detokenizing” the input [ 15 , 26 , 19 , 38 ] , it is perhaps more evocative to think of this as perception. The beginning of the model is really responsible for seeing the input, and much of the early circuitry is in service of sensing or perceiving the text similar to how early layers in vision models implement low level perception [ 54 , 39 ] . The Utility of Geometry. Many of the representations and computations we studied had elegant geometric interpretations. For example, the counting manifolds are the result of an optimal tradeoff between capacity and resolution, with deep connections to space-filling curves and Fourier features. The boundary head twist was especially beautiful, and after discovering one such head, we were able to correctly predict that there would need to be additional heads to provide curvature in the output. The distributed character counting algorithm was more complex, but we were still able to clarify our view by studying linear actions on these manifolds. For other computations, like the final breaking decision, the linear separation was clearly a part of the story but there must be some additional complexity we were not able to see yet to handle multitoken outputs. For the more semantic operations, we purely relied on the feature view. Of course, describing any behavior in full is immensely complicated, and there is a long list of possible subtleties we did not study: how the model accounts for uncertainty in its counting, its mechanism for estimating the line width given multiple prior lines of text, how it adapts to documents with variable line width, how it handles multiple plausible output tokens of different lengths or multitoken words, or various special cases (e.g., a L a T e X \footnote{} or a markdown link). For the inspired, we share transcoder attribution graphs for a fixed-width line break prompt on Gemma 2 2B and Qwen 3 4B , using the new neuronpedia interactive interface. Unsupervised Discovery It likely would not have been possible to develop this clarity if it were not for the unsupervised sparse features. In fact, when we started this project, we attempted to just probe and patch our way to understanding, but this turned out poorly. Specifically, we did not understand what we were looking for (e.g. we didn’t know to distinguish line width vs. character count), where to look for it (e.g., we didn’t expect line width to only be represented on the newline), or how to look for it (we started by training 1-D linear regression probes). However, after identifying some relevant features but before spending substantial effort systematically characterizing their activity profiles, we were also confused by what they were representing. We saw dozens of features that were vaguely about newlines and linebroken text, but their differences were not obvious from flipping through the activating examples. Only after we tested these features on synthetic datasets did their role in the graph and the underlying computation become clear. We suspect better automatic labels [ 3 , 60 , 24 ] enhanced with agentic workflows [ 65 , 4 ] would accelerate this work, especially in less verifiable domains. Feature-Manifold Duality. The discrete feature and geometric feature-manifold perspectives offer dual lenses on the same underlying object. For example, in this work the model’s representation of character count can be completely described (modulo reconstruction error) by the activities of the features we identified, where the action of the boundary heads is described by virtual weights that expand out the feature interactions via attention head matrices. The same character count representation can be described by a 1-dimensional feature manifold – a curve in the residual stream parametrized by the character count variable – where linear action of the boundary heads is described by continuous “twisting” of the manifold. In general, geometric structures learned by the model will likely admit both global parametrizations and local discrete approximations. The Complexity Tax. Despite this duality, the descriptions produced by the two perspectives differ in their simplicity. The discrete features shatter the model into many pieces, producing a complex understanding of the computation. This seems like a general lesson. It seems like discrete features and attribution graphs may provide a true description of model computation, which can be found in an automated way using dictionary learning. Getting any true, understandable description of the computation is a very non-trivial victory! However, if we stop there, and don’t understand additional structure which is present, we pay a complexity tax , where we understand things in a needlessly complicated way. In the line breaking problem, constructing the manifold paid down this tax, but one could imagine other ways of reducing the interpretation burden. A Call for Methodology. Armed with our feature understanding, we were able to directly search for the relevant geometric structures. This was an existence proof more than a general recipe, and we need methods that can automatically surface simpler structures to pay down the complexity tax. In our setting, this meant studying feature manifolds, and it would be nice to see unsupervised approaches to detecting them. In other cases we will need yet other tools to reduce the interpretation burden, like finding hierarchical representations [ 12 ] or macroscopic structure [ 56 ] in the global weights [ 2 ] . A Call for Biology. The model must perform other elegant computations. We can find these by starting with a specific task the model performs well, study this from multiple perspectives, develop methodology to answer the remaining questions, and relentlessly attempt to simplify our explanations. Because the investigation is grounded in specific examples of a behavior, it provides a fast feedback loop, can shed light on weaknesses of existing methods and inspire new ones, and can sharpen our conceptual language for understanding neural networks. We would be excited to see more deep case studies that adopt this approach. Appendix A Citation Information For attribution in academic contexts, please cite with this BibTeX citation ⬇ @article { gurnee2025when , author ={ Gurnee , Wes and Ameisen , Emmanuel and Kauvar , Isaac and Tarng , Julius and Pearce , Adam and Olah , Chris and Batson , Joshua }, title ={ When Models Manipulate Manifolds : The Geometry of a Counting Task }, journal ={ Transformer Circuits Thread }, year ={2025}, url ={ https :// transformer - circuits . pub /2025/ linebreaks / index . html } } Appendix B