HiCI: Hierarchical Construction–Integration for Long-Context Attention

Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu · 2026

Read HiCI: Hierarchical Construction–Integration for Long-Context Attention by Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20843.txt.

Introduction

Zeng and co-authors propose HiCI, a hierarchical attention module for long context language models. Most tools treat long context as a pure scaling problem. HiCI borrows from theories of how readers build discourse meaning. It first builds representations for text segments. It then merges them into a shared global context. Both levels feed back to guide attention inside each segment. Added to LLaMA-2 with few extra parameters, it stretched context to 100K tokens. Gains held across modeling, retrieval, and instruction tests.

Catalog record

CREATORXiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57632
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20843

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HiCI: Hierarchical Construction–Integration for Long-Context Attention Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu Abstract Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches. Drawing on cognitive theories of discourse comprehension, we propose HiCI (Hierarchical Construction–Integration), a hierarchical attention module that constructs segment-level representations, integrates them into a shared global context, and broadcasts both to condition segment-level attention. We validate HiCI through parameter-efficient adaptation of LLaMA-2 with only ∼ \sim 5.5% additional parameters, extending context from 4K to 100K tokens (7B) and 64K tokens (13B). Across language modeling, retrieval, and instruction-following benchmarks, HiCI yields consistent improvements over strong baselines, including matching proprietary models on topic retrieval and surpassing GPT-3.5-Turbo-16K on code comprehension. These results demonstrate the effectiveness of explicit hierarchical structuring as an inductive bias for long-context modeling. github.com/zengxyyu/HiCI 1 Introduction Large language models (LLMs) have achieved remarkable success across a wide range of natural language tasks, yet their ability to process long sequences remains fundamentally constrained by limited context windows ( 45 ; 9 ) . Long-context modeling poses two fundamental challenges: (1) efficiency —the quadratic complexity of self-attention leads to prohibitive computational and memory costs as sequence length increases; and (2) effectiveness —the ability to accept longer inputs does not necessarily yield reliable modeling of long-range dependencies ( 25 ; 32 ) . Reconciling these two requirements has emerged as a central challenge in long-context language modeling. Recent work has progressed along two complementary lines. The first pursues positional length generalization : techniques such as PI, YaRN, and PoSE ( 11 ; 37 ; 47 ) extend the usable context window by interpolating, rescaling, or simulating position indices, yet leave the attention operator—and its \mathcal{O}(n^{2}) complexity—unchanged. The second focuses on attention efficiency and architectural scalability , comprising two broad families. Sparse and grouped attention ( 8 ; 46 ; 12 ) reduces cost by restricting token connectivity, with global interactions approximated via global tokens or layer-wise multi-hop mixing from shifted grouping. Recurrent and memory-augmented architectures ( 15 ; 10 ; 36 ; 23 ) model cross-segment dependencies through compressed state propagation, but sequential processing limits parallelism and long-range information may be attenuated through the compression bottleneck. While effective for length generalization or efficiency, these approaches offer limited inductive bias for explicitly organizing long-context information into a local-to-global hierarchy that guides attention. Cognitive theories of discourse comprehension offer a principled lens on this limitation. The Construction-Integration model ( 29 ; 30 ) characterizes text understanding as a hierarchical process in which local representations are first constructed from input segments and subsequently integrated—via constraint satisfaction—into a coherent global representation. Complementarily, Global Workspace Theory ( 4 ; 17 ) posits that specialized processors operate in parallel, with information gaining access to a shared workspace being broadcast globally, achieving wide availability and exerting top-down influence on subsequent processing. This broadcast mechanism finds support in hierarchical cortical processing ( 21 ) , where top-down signals modulate lower-level representations. Taken together, these perspectives motivate a hierarchical inductive bias: local construction of segment-level representations, global integration into a shared context, and top-down broadcast to condition subsequent attention. Guided by this principle, we propose HiCI ( Hi erarchical C onstruction– I ntegration), a hierarchical attention module that instantiates construction, integration, and broadcast within Transformer attention. HiCI structures attention computation through three stages. Local construction extracts segment-level representations via cross-attention with a shared set of learnable query slots. Global integration aggregates these local representations into a compact shared context through multi-view statistical pooling and attention-based weighting. Top-down broadcast prepends both global and local representations to each segment’s key–value sequence, conditioning token-level attention on hierarchical context while preserving segment-parallel computation. We apply HiCI to pretrained LLaMA-2 models ( 43 ) , combining position interpolation ( 11 ) for context extension with FlashAttention-2 ( 16 ) for efficient long-sequence computation. Following LongLoRA’s parameter-efficient recipe ( 12 ) , we freeze the backbone and train only LoRA adapters, embedding and normalization layers, together with the proposed HiCI module. Despite adding only \sim 5.5% parameters during training, this enables context extension to 100K tokens for 7B and 64K for 13B models. At inference, HiCI is optional: it can be applied during prefill to reduce time-to-first-token latency, or omitted in favour of standard full attention. Extensive experiments on language modeling (PG-19 ( 39 ) , Proof-pile ( 3 ) ), retrieval (passkey and topic), and instruction-following (LongBench ( 7 ) ) benchmarks demonstrate that HiCI consistently improves performance over strong baselines across a wide range of tasks and context lengths. HiCI achieves 100% passkey accuracy ( 34 ) within the 32K training regime and maintains substantially higher accuracy under direct extrapolation, attains the best topic-retrieval ( 31 ) accuracy among the evaluated open-source models, and achieves higher accuracy than GPT-3.5-Turbo-16K ( 1 ) on the Code category of LongBench (+9.7%). Ablation studies further reveal two distinctive properties of hierarchical conditioning: divergent scaling with segment granularity, and near length-invariant performance under training-consistent evaluation (Std \leq 0.02), in stark contrast to shifted sparse attention used in LongLoRA (Std \geq 0.40). In summary, our contributions are: • We propose HiCI, a hierarchical attention module that instantiates construction–integration–broadcast as an explicit inductive bias for long-context modeling in Transformers. • We show that HiCI supports substantial context extension on pretrained LLaMA-2 (4K \to 100K for 7B; 4K \to 64K for 13B) with modest parameter overhead ( \sim 5.5%), yielding consistent improvements in perplexity, retrieval, and downstream tasks over strong baselines. • Systematic ablations confirm the contribution of each HiCI component and the slot capacity configuration. Under training-consistent evaluation, HiCI exhibits near length-invariant perplexity, with deeper layers increasingly attending to global representations, indicating emergent hierarchical information routing. 2 Related Work 2.1 Efficient Attention Mechanisms The quadratic complexity of self-attention has motivated extensive research on efficient alternatives. Sparse attention restricts the attention pattern to reduce computation: Longformer ( 8 ) employs sliding windows augmented with task-specific global tokens, BigBird ( 46 ) combines local, global, and random attention to achieve linear complexity with theoretical guarantees, and LongNet ( 18 ) uses dilated attention with exponentially increasing receptive fields across heads. Linear attention ( 28 ) approximates softmax via kernel decomposition, enabling O(n) complexity. However, kernel-based approximations exhibit degraded performance on retrieval-intensive tasks ( 2 ) , and predefined sparsity patterns limit adaptability to diverse long-range dependencies. 2.2 Context Window Extension for LLMs LLMs are typically pre-trained with fixed context lengths (e.g., 4,096 for Llama-2), and context extension has been pursued through positional scaling and efficient long-context adaptation . Positional encoding methods modify RoPE-style position representations to improve length extrapolation. Position Interpolation (PI) ( 11 ) rescales position indices and relies on substantial continued training to adapt to longer contexts. Subsequent schemes such as YaRN ( 37 ) and LongRoPE ( 19 ) introduce frequency-aware or non-uniform scaling, reducing the amount of long-context continued training relative to PI. These methods address where to attend but retain quadratic complexity and do not alter how attention organizes context. Training and adaptation methods address long-context fine-tuning with varying efficiency. Early work such as Focused Transformer ( 44 ) employs specialized training objectives, but remains computationally intensive (128 TPUs). More

Read the complete text