Concept Attractors in LLMs and their Applications

Sotirios Panagiotis Chytas, Vikas Singh · 2026

Read Concept Attractors in LLMs and their Applications by Sotirios Panagiotis Chytas, Vikas Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11575.txt.

Introduction

Sotirios Panagiotis Chytas and Vikas Singh explain LLM behavior with concept attractors. Their 2026 preprint uses iterated function systems for the frame. LLMs map related prompts to similar inner states. That holds at specific layers even when wording differs widely.

The team explains the pull with contractive mappings. In an iterated function system, maps shrink distances step by step. Layers then act as such maps inside the model. They draw states toward concept-specific attractors in space. An attractor is a point or set that paths fall into. This view predicts where representations settle by concept. Applications flow from knowing those stable landing points.

Catalog record

CREATORSotirios Panagiotis Chytas, Vikas Singh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11575

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Concept Attractors in LLMs and their Applications Sotirios Panagiotis Chytas, Vikas Singh Abstract Large language models (LLMs) often map semantically related prompts to similar internal representations at specific layers, even when their surface forms differ widely. We show that this behavior can be explained through Iterated Function Systems (IFS), where layers act as contractive mappings toward concept-specific Attractors. We leverage this insight and develop simple, training-free methods that operate directly on these Attractors to solve a wide range of practical tasks, including language translation , hallucination reduction , guardrailing , and synthetic data generation . Despite their simplicity, these Attractor-based interventions match or exceed specialized baselines, offering an efficient alternative to heavy fine-tuning, generalizable in scenarios where baselines underperform. 1 Introduction Figure 1: A t-sne [ 69 ] plot of the latent representations of Llama3.1-8B for 7\times 4=28 different prompts, seven each, for the Lord of the Rings universe, Narnia, Star Wars, and Harry Potter. Although the prompts explore different aspects of the universes and share almost no common keywords, we observe a clear clustering based on the different worlds. Consider three distinct concepts: the Lord of the Rings universe, the Python programming language, and 19th-century romantic literature. When prompts from these concepts are given to a large language model (LLM) such as Llama 3.1 [ 25 ] , we see an interesting phenomenon. For each concept, despite lexical variations among its prompts, their intermediate representations appear to collapse to distinct regions at specific layers – at which layer this happens varies based on the concept. For instance, prompts such as “Who is Gandalf the Grey?” and “What is the significance of Mount Doom?” share minimal similarity on the surface, yet their representations converge to nearly identical locations at layer 24. We see a similar behavior for Python-related queries such as “Help me implement a binary search tree in Python” versus “How can I find the longest non-repeating substring in Python?” and for prompts for the same genre in literature: “Discuss themes in Pride and Prejudice” and “Any easy way to recognize Byron’s poetry?”. Such a semantic collapse has been reported in some recent results. For instance, [ 62 ] notes that transformer models develop a structured latent representations that encode belief states . Separately, [ 22 ] suggests that due to the internal dynamics of the model, representations converge to “stable” configurations. From a more practical perspective, [ 28 , 46 , 63 ] showed that transformers and LLMs shape their latent space according to the underlying task. These findings, while restricted to smaller models and/or for specific contexts, cumulatively support the idea of representation collapse. A natural question is whether this concept-specific collapse is implied as a property of some underlying dynamical system already studied in the literature, and if so, what guidance can these existing results provide? Specifically, can we obtain strategies for important downstream use-cases? If p_{1},\cdots,p_{n} are a set of prompts related to a specific concept \mathcal{C} , we conjecture that the layers of our model may be acting like a dynamical system that maps semantically related inputs to proximal regions, regardless of their form at the “surface”. In other words, the full sequence of layers (leading up to where the representations collapse), if viewed as a unit, implements an iterative (contractive) mapping process to an Attractor set , one for each concept. We will see shortly that – to the extent that our hypothesis holds – how existing results are consistent with this view of the collapse phenomena. Contributions. We show that viewing the LLMs through the lens of Iterated Function Systems [ 4 , 32 ] offers a meaningful (or at worst, plausible) explanation for both the layer-specific concept clustering and the subsequent generative process. The main practical benefit is that for a wide-variety of downstream tasks, which are often handled piecemeal in the literature, we can obtain a generic scheme that operates under the assumption that operating with the Attractors alone is sufficient . We demonstrate that careful interventions on Attractors can provide us lightweight, training-free solutions to a wide array of problems, from programming language translation and guardrailing , to hallucination reduction and synthetic data generation . Despite the simplicity as well as limited data/compute needs, these solutions turn out to be comparable to existing specialized approaches. Our experiments focus on Llama3.1 8B [ 25 ] . However, we see a similar behavior on other LLM families too (in particular, Gemma [ 66 ] and Qwen [ 67 ] ), but avoid an exhaustive analysis of all LLMs. 2 Iterated Function Systems and LLMs There is mounting evidence that large language models (LLMs) possess emergent capabilities beyond simple rote memorization and statistical pattern matching [ 6 ] . Among the many phenomena observed in these models – from in-context learning [ 16 ] to compositional reasoning [ 53 , 40 ] – we focus on a particular representation-convergence property. Our scope is specifically the collapse phenomena at specific intermediate layers. To understand this behavior through the lens of dynamical systems, we hypothesize that LLMs implicitly implement a collection of Iterated Function Systems (IFS) during forward propagation through the layers (Fig. 2 ). 2.1 LLMs implement Iterated Function Systems? Empirically, we see that for prompts p_{i} , p_{j} in each concept \mathcal{C} , there exists a layer l where: \lim_{l\rightarrow l_{\mathcal{C}}}\frac{1}{n^{2}}\sum_{i,j=1}^{n}|h_{l}(p_{i})-h_{l}(p_{j})|\ll\frac{1}{n^{2}}\sum_{i,j=1}^{n}|h_{0}(p_{i})-h_{0}(p_{j})| (1) Figure 2: An LLM can be viewed as an IFS that transforms the non-linear manifold of texts into a well-behaving collection of Attractors. with h_{l} denoting the implicit transformation by the LLM up to layer l . This “squashing” of inter-prompt distances suggests that a contractive mapping process is taking place through the layers. Our hypothesis is that this can be understood via the framework of Iterated Function Systems (IFS) [ 4 , 32 ] . An IFS is defined as a finite set of contractive mappings on a complete metric space. The collective action of these mappings, defined by the Hutchinson operator [ 32 ] is: \footnotesize\mathcal{F}(\mathbf{S})=\bigcup_{i=1}^{N}f_{i}(\mathbf{S}) (2) and induces a compact invariant set i.e., \mathcal{F}(\mathbf{S}^{*})=\mathbf{S}^{*} , which is called the Attractor of the IFS. More generally, for any initial non-empty compact set \mathbf{S}_{0}\in\mathbb{X} , the sequence \{\mathbf{S}_{0},\mathbf{S}_{1}\coloneqq\mathcal{F}(\mathbf{S}_{0}),\mathbf{S}_{2}\coloneqq\mathcal{F}(\mathbf{S}_{1}),\cdots\} converges to \mathbf{S}^{*} in the Haussdorf metric. More generally, an Attractor in a dynamical system is a closed invariant set toward which trajectories from a wide class of initial conditions evolve asymptotically within its basin of attraction, and may take the form of fixed points, periodic orbits, tori, or other Attractors characterized by sensitive dependence on initial conditions [ 4 ] . Dynamical systems often exhibit Attractors—sets toward which trajectories converge. Simple systems satisfying Banach’s fixed-point conditions [ 3 ] converge to a single point, while others yield more complex structures like limit cycles or strange Attractors [ 65 ] . We hypothesize that the iterative application of layer transformations in an LLM induces concept-specific invariant sets –semantic Attractors ( \mathbf{A}^{\mathcal{C}}_{l} ) for each concept \mathcal{C} – within the latent space at layer l . These compact regions characterize specific concepts, with convergence potentially occurring at different depths depending on the concept. Once a sequence’s representation enters \mathbf{A}^{\mathcal{C}}_{l} , it is further processed by the remaining layers and output matrix W_{\rm

Read the complete text