HiCI: Hierarchical Construction–Integration for Long-Context Attention
Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu · 2026
Read HiCI: Hierarchical Construction–Integration for Long-Context Attention by Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20843.txt.
Introduction
Zeng and co-authors propose HiCI, a hierarchical attention module for long context language models. Most tools treat long context as a pure scaling problem. HiCI borrows from theories of how readers build discourse meaning. It first builds representations for text segments. It then merges them into a shared global context. Both levels feed back to guide attention inside each segment. Added to LLaMA-2 with few extra parameters, it stretched context to 100K tokens. Gains held across modeling, retrieval, and instruction tests.
Catalog record
| CREATOR | Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57632 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20843
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HiCI: Hierarchical Construction–Integration for Long-Context Attention Xiangyu Zeng, Qi Xu, Yunke Wang, Chang Xu Abstract Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches. Drawing on cognitive theories of discourse comprehension, we propose HiCI (Hierarchical Construction–Integration), a hierarchical attention module that constructs segment-level representations, integrates them into a shared global context, and broadcasts both to condition segment-level attention. We validate HiCI through parameter-efficient adaptation of LLaMA-2 with only ∼ \sim 5.5% additional parameters, extending context from 4K to 100K tokens (7B) and 64K tokens (13B). Across language modeling, retrieval, and instruction-following benchmarks, HiCI yields consistent improvements over strong baselines, including matching proprietary models on topic retrieval and surpassing GPT-3.5-Turbo-16K on code comprehension. These results demonstrate the effectiveness of explicit hierarchical structuring as an inductive bias for long-context modeling. github.com/zengxyyu/HiCI 1 Introduction Large language models (LLMs) have achieved remarkable success across a wide range of natural language tasks, yet their ability to process long sequences remains fundamentally constrained by limited context windows ( 45 ; 9 ) . Long-context modeling poses two fundamental challenges: (1) efficiency —the quadratic complexity of self-attention leads to prohibitive computational and memory costs as sequence length increases; and (2) effectiveness —the ability to accept longer inputs does not necessarily yield reliable modeling of long-range dependencies ( 25 ; 32 ) . Reconciling these two requirements has emerged as a central challenge in long-context language modeling. Recent work has progressed along two complementary lines. The first pursues positional length generalization : techniques such as PI, YaRN, and PoSE ( 11 ; 37 ; 47 ) extend the usable context window by interpolating, rescaling, or simulating position indices, yet leave the attention operator—and its \mathcal{O}(n^{2}) complexity—unchanged. The second focuses on attention efficiency and architectural scalability , comprising two broad families. Sparse and grouped attention ( 8 ; 46 ; 12 ) reduces cost by restricting token connectivity, with global interactions approximated via global tokens or layer-wise multi-hop mixing from shifted grouping. Recurrent and memory-augmented architectures ( 15 ; 10 ; 36 ; 23 ) model cross-segment dependencies through compressed state propagation, but sequential processing limits parallelism and long-range information may be attenuated through the compression bottleneck. While effective for length generalization or efficiency, these approaches offer limited inductive bias for explicitly organizing long-context information into a local-to-global hierarchy that guides attention. Cognitive theories of discourse comprehension offer a principled lens on this limitation. The Construction-Integration model ( 29 ; 30 ) characterizes text understanding as a hierarchical process in which local representations are first constructed from input segments and subsequently integrated—via constraint satisfaction—into a coherent global representation. Complementarily, Global Workspace Theory ( 4 ; 17 ) posits that specialized processors operate in parallel, with information gaining access to a shared workspace being broadcast globally, achieving wide availability and exerting top-down influence on subsequent processing. This broadcast mechanism finds support in hierarchical cortical processing ( 21 ) , where top-down signals modulate lower-level representations. Taken together, these perspectives motivate a hierarchical inductive bias: local construction of segment-level representations, global integration into a shared context, and top-down broadcast to condition subsequent attention. Guided by this principle, we propose HiCI ( Hi erarchical C onstruction– I ntegration), a hierarchical attention module that instantiates construction, integration, and broadcast within Transformer attention. HiCI structures attention computation through three stages. Local construction extracts segment-level representations via cross-attention with a shared set of learnable query slots. Global integration aggregates these local representations into a compact shared context through multi-view statistical pooling and attention-based weighting. Top-down broadcast prepends both global and local representations to each segment’s key–value sequence, conditioning token-level attention on hierarchical context while preserving segment-parallel computation. We apply HiCI to pretrained LLaMA-2 models ( 43 ) , combining position interpolation ( 11 ) for context extension with FlashAttention-2 ( 16 ) for efficient long-sequence computation. Following LongLoRA’s parameter-efficient recipe ( 12 ) , we freeze the backbone and train only LoRA adapters, embedding and normalization layers, together with the proposed HiCI module. Despite adding only \sim 5.5% parameters during training, this enables context extension to 100K tokens for 7B and 64K for 13B models. At inference, HiCI is optional: it can be applied during prefill to reduce time-to-first-token latency, or omitted in favour of standard full attention. Extensive experiments on language modeling (PG-19 ( 39 ) , Proof-pile ( 3 ) ), retrieval (passkey and topic), and instruction-following (LongBench ( 7 ) ) benchmarks demonstrate that HiCI consistently improves performance over strong baselines across a wide range of tasks and context lengths. HiCI achieves 100% passkey accuracy ( 34 ) within the 32K training regime and maintains substantially higher accuracy under direct extrapolation, attains the best topic-retrieval ( 31 ) accuracy among the evaluated open-source models, and achieves higher accuracy than GPT-3.5-Turbo-16K ( 1 ) on the Code category of LongBench (+9.7%). Ablation studies further reveal two distinctive properties of hierarchical conditioning: divergent scaling with segment granularity, and near length-invariant performance under training-consistent evaluation (Std \leq 0.02), in stark contrast to shifted sparse attention used in LongLoRA (Std \geq 0.40). In summary, our contributions are: • We propose HiCI, a hierarchical attention module that instantiates construction–integration–broadcast as an explicit inductive bias for long-context modeling in Transformers. • We show that HiCI supports substantial context extension on pretrained LLaMA-2 (4K \to 100K for 7B; 4K \to 64K for 13B) with modest parameter overhead ( \sim 5.5%), yielding consistent improvements in perplexity, retrieval, and downstream tasks over strong baselines. • Systematic ablations confirm the contribution of each HiCI component and the slot capacity configuration. Under training-consistent evaluation, HiCI exhibits near length-invariant perplexity, with deeper layers increasingly attending to global representations, indicating emergent hierarchical information routing. 2 Related Work 2.1 Efficient Attention Mechanisms The quadratic complexity of self-attention has motivated extensive research on efficient alternatives. Sparse attention restricts the attention pattern to reduce computation: Longformer ( 8 ) employs sliding windows augmented with task-specific global tokens, BigBird ( 46 ) combines local, global, and random attention to achieve linear complexity with theoretical guarantees, and LongNet ( 18 ) uses dilated attention with exponentially increasing receptive fields across heads. Linear attention ( 28 ) approximates softmax via kernel decomposition, enabling O(n) complexity. However, kernel-based approximations exhibit degraded performance on retrieval-intensive tasks ( 2 ) , and predefined sparsity patterns limit adaptability to diverse long-range dependencies. 2.2 Context Window Extension for LLMs LLMs are typically pre-trained with fixed context lengths (e.g., 4,096 for Llama-2), and context extension has been pursued through positional scaling and efficient long-context adaptation . Positional encoding methods modify RoPE-style position representations to improve length extrapolation. Position Interpolation (PI) ( 11 ) rescales position indices and relies on substantial continued training to adapt to longer contexts. Subsequent schemes such as YaRN ( 37 ) and LongRoPE ( 19 ) introduce frequency-aware or non-uniform scaling, reducing the amount of long-context continued training relative to PI. These methods address where to attend but retain quadratic complexity and do not alter how attention organizes context. Training and adaptation methods address long-context fine-tuning with varying efficiency. Early work such as Focused Transformer ( 44 ) employs specialized training objectives, but remains computationally intensive (128 TPUs). More