Change of Thought: Adaptive Test-Time Computation
Mrinal Mathur, Mike Doan, Barak A. Pearlmutter, Sergey Plis · 2025
Read Change of Thought: Adaptive Test-Time Computation by Mrinal Mathur, Mike Doan, Barak A. Pearlmutter, Sergey Plis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13569.txt.
Introduction
Mrinal Mathur and co-authors ask how a transformer model can think harder on hard inputs. A plain transformer reads its input once, in a single pass of fixed depth. That design caps what the model can express. Chain of thought methods get past the cap, but they spell out each step in words and read those words back in.
The authors offer a new design, the SELF-Transformer, that skips the word loop. Its layers adjust their own attention weights again and again until they settle. Hard inputs thus get more work at test time than easy ones. Tests on encoder tasks show gains of up to twenty percent, with no added parameters and only a modest rise in cost.
Catalog record
| CREATOR | Mrinal Mathur, Mike Doan, Barak A. Pearlmutter, Sergey Plis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-13569
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Change of Thought: Adaptive Test-Time Computation Mrinal Mathur, Mike Doan, Barak A. Pearlmutter, Sergey Plis Abstract Transformers evaluated in a single, fixed-depth pass are provably limited in expressive power to the constant-depth circuit class T C ∼ 0 TC\sim 0 . Running a Transformer autoregressively removes that ceiling—first in next-token prediction and, more recently, in chain-of-thought reasoning. Both regimes rely on feedback loops that decode internal states into tokens only to re-encode them in subsequent steps. While this “thinking aloud” mirrors human reasoning, biological brains iterate without externalising intermediate states as language. To boost the expressive power of encoder Transformers without resorting to token-level autoregression, we introduce the SELF-Transformer: an encoder layer that iteratively refines its own attention weights to a fixed point. Instead of producing—in one pass—the alignment matrix that remixes the input sequence, the SELF-Transformer iteratively updates that matrix internally, scaling test-time computation with input difficulty. This adaptivity yields up to 20% accuracy gains on encoder-style benchmarks without increasing parameter count, demonstrating that input-adaptive alignment at test time offers substantial benefits for only a modest extra compute budget. Self-Transformers thus recover much of the expressive power of iterative reasoning while preserving the simplicity of pure encoder architectures. 1 Introduction The advent of transformers ( vaswani2017attention ) has revolutionized natural language processing (NLP) and computer vision, achieving state-of-the-art results in language modeling, question answering, and image classification. At the core of their success lies the self-attention mechanism, which models long-range dependencies by dynamically computing input-dependent alignment matrices that determine how input tokens or patches are mixed. Transformers have achieved their widest success as large language models (LLM) through autoregressive operation that enabled success of large language models, first in next-token prediction ( brown2020language ) and more recently in chain-of-thought reasoning ( wei2022chain ) . However, their reliance on token-level autoregression introduces inefficiencies: transformers repeatedly decode internal states into tokens and re-encode them in subsequent steps, creating computational bottlenecks. Unlike LLMs, the encoder transformers—the focus of this work— do not use causal attention and are widely used across a wide range of applications and models dosovitskiy2020image ; liu2021swin ; radford2018improving ; brown2020language ; Radford2021CLIP . It is unclear whether autoregressive trick can be used to improve performance of these models. In common use, transformers remain powerful but rigid feedforward networks, that benefit from the external loop of their operation. To address these inefficiencies, recent work has explored dynamic computation networks ( han2021dynamic ) , memory-augmented architectures ( khandelwal2021nearest ; rae2019compressive ) , and other mechanisms, as discussed in detail in Section 2 . However, these approaches often trade simplicity for performance by introducing complex hypernetworks or external retrieval modules that inflate parameter counts and computational costs. Furthermore, while alignment matrices in conventional transformers are input-dependent and should account for input complexity, they are computed in a single feed-forward pass. This limits their ability to dynamically refine generated representations based on the obtained output. Figure 1 : SELF-Transformer modifies vanilla self-attention (A) to iteratively update its alignment transform {\cal T} , adapting to the input sequence without introducing additional parameters (B). Compare with dynamic computation (C), which applies the same transformer block sequentially multiple times, and existing fixed-point iteration (D), which iterates the block to a fixed point. In this work, we propose the Self-Enhancing Latent Feedback Transformer (SELF-Transformer), a novel architecture that augments self-attention (a) with Fixed-Point Iteration (FPI) to enable latent alignment refinement ( Fig. 1 ). Instead of generating alignment matrices in a single forward pass, SELF-Transformer iteratively updates them within the self-attention module. This iterative refinement process allows the model to dynamically adjust its attention patterns based on input complexity, spending more computational effort on challenging representations while maintaining efficiency for simpler inputs. By leveraging FPI universally across all layers, SELF-Transformer achieves deeper contextual reasoning without introducing additional parameters or task-specific modifications. Crucially, this approach enables SELF-Transformer to autoregressively improve its latent representations at each layer without the need to first decode them into tokens, all while incurring zero additional parameters. An important property of Fixed-Point Self-Attention (FPSA) Fig. 1 B is its focus on the iterative refinement of the layer’s parameters—the alignment matrix—rather than the output, which is more common in dynamic computation approaches ( Fig. 1 C&D). Although the end result similarly leads to an improvement in the output of the dynamic layer, FPSA explicitly operates on the dynamic parameters of that layer. Exploiting fixed-point iterations simplifies both the forward and backward paths during training, eliminating memory overhead during the process. This avoids the need for unrestricted memory consumption or checkpointing, which are often required in other dynamic computation methods. 2 Related Work Dynamic Computation in Neural Networks The concept of dynamically adjusting computational effort based on input complexity has been a longstanding focus in machine learning. Early work by ( graves2016adaptive ) introduced Adaptive Computation Time (ACT) for recurrent networks, allowing models to determine the number of computation steps per input. Stochastic depth networks ( huang2016deep ) and early-exit architectures ( teerapittayanon2016branchynet ) later demonstrated that skipping layers or operations could improve efficiency without sacrificing accuracy. More recently, ( banino2021pondernet ) formalized adaptive computation using learned halting distributions, while ( raposo2024mixture ) proposed Mixture-of-Depths (MoD) to dynamically allocate compute across transformer layers. The Dynamic Diffusion Transformer (DyDiT) ( zhao2024dynamic ) further extends these ideas by dynamically adjusting computation along both timestep and spatial dimensions in diffusion models, achieving significant efficiency gains (see Fig. 1 C for an example). Unlike these approaches, which often rely on auxiliary networks or heuristics, our method applies fixed-point iteration (FPI) universally within each layer as shown in Fig. 1 B, enabling fine-grained refinement of self-attention alignments without introducing additional parameters. Fixed-Point Methods in Deep Learning Fixed-point iteration has emerged as a powerful tool for modeling implicit depth in neural networks. Building on foundational work ( FEYNMAN39A ; almeida1987learning ; pineda1987generalization ) , and their more modern revival ( pmlr-v80-liao18c ) methods like Deep Equilibrium Models (DEQs) ( bai2019deep ) define infinite-depth networks through fixed-point equations. However, DEQs face challenges such as slow convergence and sensitivity to initialization ( geng2021training ) , limiting their practicality. Recent advances address these issues via phantom gradients ( jeon2021differentiable ) and nonsmooth optimization techniques ( barrett2024invariant ) . Complementary to these works, ( ke2024advancing ) conducted a detailed analysis of fixed-point iterations in high-dimensional neural networks, identifying conditions for convergence and stability. Our work diverges by applying FPI explicitly to self-attention matrices rather than complete layers, enabling iterative refinement of token alignments while maintaining gradient stability through implicit differentiation ( bolte2022automatic ) as shown in Fig. 1 (B). This bridges the gap between fixed-point theory and transformer architectures, offering a principled alternative to ad hoc depth adaptation. Adaptive Transformers Resource-efficient transformers have gained significant attention due to their ability to dynamically adjust computation. Spatially Adaptive Computation Time (SACT) was first proposed for CNNs by ( figurnov2017spatially ) and later extended to transformers by ( elbayad2019depth ) . Meanwhile, ( wang2024augmenting ) augmented LLMs with external memory modules for long-context tasks at the cost of inflated parameter counts. Closest to our work, Mixture-of-Depths (MoD) transformers ( raposo2024mixture ) and Enhanced Transformers with dynamic token routing ( yang2024enhancing ) activate subsets of layers or attention