When to Think Fast and Slow? AMOR: Entropy-Based Metacognitive Gate for Dynamic SSM-Attention Switching

Haoran Zheng · 2026

Read When to Think Fast and Slow? AMOR: Entropy-Based Metacognitive Gate for Dynamic SSM-Attention Switching by Haoran Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-13215.txt.

Introduction

Hybrid looping-focus models promise recurrence leanness plus focus recall, but most apply focus uniformly, even where recurrence alone would suffice. Haoran Zheng and Chen Shani present AMOR, an adaptive metacognitive output router that invokes focus selectively from predictive doubt. A looping backbone gains mess-gated focus blocks that switch on only when output mess passes a dynamic threshold set from a running batch median plus scaled standard deviation. It needs no routing settings to learn.

Training from scratch on FineWeb-Edu with focus on only about 40% of positions, an AMOR variant leads its scale on eight-task commonsense thinking. It beats pure looping models, pure focus models, and fixed-schedule hybrids. Pull lifts over pure looping models, stays competitive with fixed-schedule hybrids, and keeps looping backbones' long-context toughness where transformers degrade under spread shift. The lesson is timing: selective focus by doubt lifts closeness, toughness, and leanness.

Catalog record

CREATORHaoran Zheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-13215

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When to Think Fast and Slow? AMOR: Entropy-Based Metacognitive Gate for Dynamic SSM-Attention Switching Haoran Zheng Abstract Transformers allocate uniform computation to every position, regardless of difficulty. State Space Models (SSMs) offer efficient alternatives but struggle with precise information retrieval over a long horizon. Inspired by dual-process theories of cognition ( 16 ) , we propose Amor (Adaptive Metacognitive Output Router), a hybrid architecture that dynamically engages sparse attention only when an SSM backbone is “uncertain”—as measured by prediction entropy. Compared to standard transformers, Amor gains efficiency by projecting keys and values from SSM hidden states (Ghost KV), reusing the SSM’s O ⁡ ( n ) O(n) computation rather than requiring O ⁡ ( n 2 ) O(n^{2}) attention at every layer. On small-scale synthetic retrieval tasks, Amor outperforms both SSM-only and transformer-only baselines, achieving perfect retrieval accuracy while engaging attention on only 22% of positions. We validate that prediction entropy reliably signals retrieval need, with a gap of 1.09 nats (nearly half the entropy range) between retrieval and local positions. Additionally, our approach provides interpretable adaptive computation, where routing decisions can be understood in information-theoretic terms. 1 Introduction Transformers ( 25 ) have become the dominant architecture for sequence modeling, achieving strong results across language, vision, and multimodal domains. Yet they allocate identical computation to every position in a sequence, whether predicting a common word from obvious context or retrieving a specific fact from long-range memory. This uniform allocation ignores that prediction difficulty varies across positions—some require only local context while others demand retrieval from distant memory. Human cognition suggests an alternative. 16 distinguished between System 1 (fast, automatic, effortless) and System 2 (slow, deliberate, effortful) thinking. For instance, reading familiar prose is System 1: predictions flow automatically from context. Encountering an unfamiliar technical term triggers System 2: we pause, perhaps re-read, and retrieve relevant background. Notably, this switching is metacognitive : humans recognize uncertainty and engage deliberate processing accordingly. 5 suggested that neural networks might benefit from similar dual-process mechanisms, allocating deliberate computation where automatic processing falls short. State Space Models (SSMs) offer a promising substrate for mimicking efficient System 1 processing. Mamba ( 14 ) , for instance, achieves linear complexity by compressing sequence history into a fixed-size recurrent state, excelling at local pattern modeling. But this compression comes at a cost: precise information from distant positions cannot always be recovered from a compressed state. Hybrid architectures like Jamba ( 19 ) and Griffin ( 7 ) emerged to address this tradeoff. These designs combine SSMs with attention, but still apply attention uniformly across sequences, without adapting to which positions actually need it. Adaptive computation methods address this by varying computation based on input. Mixture-of-Depths ( 21 ) learns to route tokens around computation entirely, while PonderNet ( 2 ) learns when to stop iterating on a representation. These approaches achieve efficiency gains, but through learned routers whose decisions resist interpretation. Given a position that receives extra computation, we cannot easily say why . Moreover, they vary the amount of computation, not the type: there is no qualitative shift between processing modes as dual-process theory suggests. This Work. We propose Amor (Adaptive Metacognitive Output Router) 1 1 1 Code and replication materials available at https://github.com/HaoranZhengRaul/AMOR , where an SSM backbone processes all positions, and attention is selectively engaged based on a simple, interpretable signal: prediction entropy. The core insight is that entropy measures uncertainty. When the SSM’s prediction distribution is concentrated—low entropy—local context suffices. When the distribution is diffuse—high entropy—the model is uncertain, and calls in attention to help. Figure 1 : Amor Architecture. An SSM (System 1) processes each token, producing predictions and hidden states. The entropy gate monitors prediction entropy: high entropy (”I don’t know”) triggers sparse attention (System 2) over the Ghost KV cache—keys and values projected from SSM hidden states, providing temporally-aware representations for retrieval. Amor combines three components (Figure 1 ): • System 1 (SSM): Processes all positions with linear complexity, producing predictions and hidden states • Entropy Gate: Triggers attention when prediction entropy exceeds a learned threshold • System 2 (Sparse Attention): Retrieves over a Ghost KV cache—keys and values projected from SSM hidden states rather than raw embeddings Table 1 : Per-token entropy gating (toy example). High-entropy positions (3, 7) trigger attention. Position 1 2 3 4 5 6 7 8 9 Entropy 0.2 0.3 1.9 0.4 0.2 0.3 2.1 0.2 0.3 Gate 0 0 1 0 0 0 1 0 0 This design has several appealing properties. The routing signal is interpretable : we can inspect entropy at each position and understand, in information-theoretic terms, why attention was triggered. Entropy is principled : it measures genuine model uncertainty rather than black-box routing patterns that may not generalize. And the Ghost KV cache reuses the SSM’s already-computed O(n) hidden states rather than computing fresh O(n^{2}) attention representations for every position, achieving efficiency gains over standard transformers. 2 Related Work State Space Models. Structured State Space Models (S4) ( 15 ) introduced efficient long-range sequence modeling through continuous-time formulations, achieving strong results on the Long Range Arena benchmark ( 24 ) . Mamba ( 14 ) extended this line of work with selective state spaces that condition parameters on input content, enabling content-based reasoning while maintaining linear complexity. The recent Mamba-2 ( 6 ) established theoretical connections between SSMs and attention through structured state space duality, revealing that these seemingly distinct architectures share deep mathematical structure. Despite these advances, SSMs fundamentally compress sequence history into fixed-size state, which limits their ability to retrieve precise information from arbitrary past positions. Hybrid Architectures. Several architectures combine SSMs with attention to leverage both efficiency and retrieval capability. Jamba ( 19 ) interleaves Mamba and attention layers at a fixed 1:7 ratio, incorporating Mixture-of-Experts for additional capacity. Griffin ( 7 ) combines gated linear recurrences with local sliding-window attention, achieving strong performance on language modeling while reducing the quadratic cost of global attention. RWKV ( 20 ) reformulates attention as a linear recurrence, enabling RNN-style inference with transformer-quality results. More recent work explores different fusion strategies: Hymba ( 9 ) fuses attention and SSM heads in parallel within each layer, using learned meta tokens to reduce attention burden, while Zamba ( 12 ) takes a minimal approach with a Mamba backbone and single shared attention layer. These architectures use static patterns of interleaving—every sequence receives the same architectural treatment regardless of content. Amor differs in two ways: the SSM processes all positions while attention is selectively engaged only where the model’s own uncertainty signals retrieval is needed, and the routing decision is itself interpretable through entropy—a metacognitive signal that reveals why additional computation was allocated. Adaptive Computation. The idea of varying computation based on input difficulty has a rich history. Adaptive Computation Time (ACT) ( 13 ) pioneered variable-depth computation in RNNs by learning a halting mechanism. Universal Transformers ( 8 ) applied ACT to transformers, iterating over representations until a learned halting condition is met. PonderNet ( 2 ) improved on ACT with a probabilistic halting formulation that yields unbiased gradients. These approaches vary computation depth —how many times a representation is refined. Amor instead varies computation type at fixed depth: certain positions receive

Read the complete text