SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling

Krishna C. Puvvada, Faisal Ladhak, Santiago Akle Serrano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, Boris Ginsburg, NVIDIA · 2025

Read SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling by Krishna C. Puvvada, Faisal Ladhak, Santiago Akle Serrano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, Boris Ginsburg, NVIDIA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08719.txt.

Introduction

Krishna Puvvada and co-authors present a 2025 design for language models that have to read very long texts. Standard models often fail when a text runs far past the length seen in training. Long inputs also cost much compute time and money.

Their model, SWAN-GPT, mixes two kinds of layers in one stack. Some layers use no position codes at all, while others use a sliding window with rotary codes. A simple scaling of attention scores at run time helps the model stretch to longer inputs. Tests showed strong results on texts far longer than training texts, with lower cost. Older trained models can also be converted to the new design with brief extra training.

Catalog record

CREATORKrishna C. Puvvada, Faisal Ladhak, Santiago Akle Serrano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, Boris Ginsburg, NVIDIA
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08719

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling Krishna C. Puvvada, Faisal Ladhak, Santiago Akle Serrano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, Boris Ginsburg, NVIDIA Abstract We present a decoder-only Transformer architecture that robustly generalizes to sequence lengths substantially longer than those seen during training. Our model, SWAN-GPT, interleaves layers without positional encodings (NoPE) and sliding-window attention layers equipped with rotary positional encodings (SWA-RoPE). Experiments demonstrate strong performance on sequence lengths significantly longer than the training length without the need for additional long-context training. This robust length extrapolation is achieved through our novel architecture, enhanced by a straightforward dynamic scaling of attention scores during inference. In addition, SWAN-GPT is more computationally efficient than standard GPT architectures, resulting in cheaper training and higher throughput. Further, we demonstrate that existing pre-trained decoder-only models can be efficiently converted to the SWAN architecture with minimal continued training, enabling longer contexts. Overall, our work presents an effective approach for scaling language models to longer contexts in a robust and efficient manner. 1 Introduction Large Language Models based on standard decoder-only transformer architectures [ 6 , 17 , 42 ] struggle with context lengths beyond their training distribution. Current approaches to extending context length fall into two categories: specialized training on increasingly longer sequences [ 17 , 42 , 33 , 7 ] or complex inference time modifications [ 1 ] . These approaches incur either increased computation cost or increased implementation complexity. We propose SWAN-GPT, a decoder-only transformer architecture that natively handles sequences substantially longer than seen during training without requiring additional long-context-specific training. By strategically interleaving global attention layers without positional encodings and local, sliding-window attention layers with rotary position encodings, combined with a dynamic attention scaling mechanism, SWAN-GPT achieves both remarkable length generalization and significant computational efficiency. This architecture not only maintains comparable performance to standard transformers on established LLM benchmarks, but also achieves robust extrapolation to sequences far beyond the training length, providing a more scalable and efficient solution to the long-context challenge. A central challenge in extending transformer context lengths is the handling of positional information. Transformers rely on positional encodings to track token order, but these encodings often become unreliable when models process sequences longer than those seen during training. Among the various positional encoding schemes, Rotary Positional Encodings (RoPE) [ 36 ] have been widely adopted in modern language models due to effectiveness in capturing relative positions. However, RoPE-based models exhibit significant performance degradation when applied to sequences exceeding their training length. This degradation occurs because inter-token distances advance to ranges where the relative rotation angle is outside the trained distribution [ 27 ] . To address this limitation, we explore two complementary approaches with distinct strengths and limitations. Sliding window attention with RoPE (SWA-RoPE) restricts every token’s attention to a fixed-size window of neighboring tokens. Because the distance between attended tokens is bounded, SWA-RoPE layers never operate at rotation angles outside their training range, making them inherently robust to arbitrary sequence lengths. However, this locality constraint limits their ability to capture long-range dependencies. Conversely, layers without positional encoding (NoPE) [ 20 , 24 ] allow unrestricted attention across the entire context while omitting explicit positional information. Notably, autoregressive NoPE models can develop implicit positional awareness through the causal attention mask, achieving comparable perplexity to models with explicit positional embeddings [ 20 ] . Despite this capability, pure NoPE models also exhibit poor robustness beyond their training length, with performance degrading rapidly due to the brittleness of the learned positional mechanism. Our key insight is that these approaches can complement each other through strategic integration. SWAN-GPT interleaves global attention layers without positional encodings (NoPE) and local sliding-window attention layers with rotary positional encodings (SWA-RoPE). This hybrid design creates a synergistic effect: SWA-RoPE layers provide local positional structure, while NoPE layers integrate information across arbitrary distances. When interleaved, the NoPE layers develop more robust representations than they would in isolation, enabling the entire model to generalize beyond its training sequence length. Unlike standard RoPE-based transformers which experience catastrophic performance collapse outside their training context, SWAN maintains robust performance on extended sequences with only a straightforward rescaling of attention scores during inference. In Section 2.1, we provide evidence that failures in the implicit position prediction mechanism of NoPE models contribute to their performance degradation on longer sequences, and demonstrate how the interleaved SWA-RoPE layers stabilize this mechanism. Additionally, we show that existing transformer models can be efficiently adapted to the SWAN architecture through continued pretraining (CPT), offering a practical, cost-effective path to upgrading deployed models. Our contributions are as follows: 1. A novel architecture (SWAN) that combines SWA-RoPE and NoPE layers to enable efficient length extrapolation without additional training, enhanced by a logarithmic attention scaling mechanism for inference. 2. Mechanistic analysis explaining why this architecture produces robust length extrapolation, with evidence that NoPE layers develop more stable positional encodings when paired with SWA-RoPE layers. 3. Empirical results demonstrating that SWAN maintains robust performance on sequences far exceeding its training length, while achieving comparable results to standard transformer architectures on established LLM benchmarks. 4. A practical method for adapting existing transformer models to the SWAN architecture through continued pre-training (CPT), providing a cost-effective upgrade path for deployed models. 2 The SWAN-GPT architecture SWAN-GPT is a decoder-only Transformer architecture that addresses the challenge of length extrapolation by interleaving two types of attention mechanisms: global attention layers without positional encodings (NoPE) and local sliding-window attention layers with rotary positional encodings (SWA-RoPE). This hybrid design leverages the complementary strengths of both approaches to achieve robust length extrapolation capabilities, without specialized long-context training. As detailed in our ablation studies ( Appendix A ), we explored multiple configurations for interleaving these layer types. Our experiments revealed that beginning with a global NoPE layer followed by three consecutive sliding-window layers, repeating this pattern throughout the network, demonstrated superior performance on long-context tasks. This configuration achieves exceptional NIAH scores at context lengths 16 times longer than the training length, and maintains robust performance even at 32 times the training length when combined with appropriate attention scaling ( subsection 2.2 ). We adopt this interleaving pattern for all experiments presented in the main paper. The global NoPE layers permit unrestricted attention across the entire context, enabling the model to capture long-range dependencies. Meanwhile, the local SWA-RoPE layers operate with a fixed window size of 512 tokens, providing consistent positional information within a bounded context. This architecture creates a complementary system where SWA-RoPE layers enforce local positional structure while NoPE layers integrate information across arbitrary distances. The key insight is that when these mechanisms are interleaved, the NoPE layers develop more robust position-aware representations than they would in isolation, enabling the entire model to generalize effectively beyond training sequence lengths. Figure 1: Mean negative log likelihood by token position for GPT with rotary positional encodings (RoPE GPT, blue), a GPT with no positional encodings (NoPE, orange), a Swan model (red), and a model conposed only of sliding window attention layers (SWA, green). Both RoPE GPT and NoPE models struggle beyond training sequence length (1024). SWA model doesn’t experience such catastrophic failure due to its limited context. Swan model behaves like a

Read the complete text