Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing

Jinchang Zhu, Jindong Li, Chengyu Zou, Rong Fu, Chao Wang, Haowei He, Menglin Yang, [0.3em] The Hong Kong University of Science, Technology (Guangzhou), [0.2em] [email protected], [0.2em] Corresponding author · 2026

Read Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing by Jinchang Zhu, Jindong Li, Chengyu Zou, Rong Fu, Chao Wang, Haowei He, Menglin Yang, [0.3em] The Hong Kong University of Science, Technology (Guangzhou), [0.2em] [email protected], [0.2em] Corresponding author (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10544.txt.

Introduction

Jinchang Zhu and six colleagues reframe long-context tuning as a training problem. Long-context tuning is commonly treated as window scaling: raise training sequence length, keep pretraining, and expect better long-context behavior. That view misses a token-level training mismatch. Longer windows allow longer possible contexts, while causal modeling delivers training one target token at a time. Their 2026 paper introduces EXACT.

EXACT, or Effective-context Allocation for Context Training, measures the exposure spread across real left contexts. It assigns a fixed extra training weight to long real-context targets, distributed by inverse frequency within the long-context tail. Across seven Qwen and LLaMA training setups, EXACT improves all twenty-eight trained and unseen NoLiMa and RULER comparisons. On Qwen2.5-0.5B, it improves NoLiMa by 10.09 points in trained lengths and 5.34 in unseen lengths. Standard QA and reasoning hold steady, with a 0.24 macro change across six benchmarks.

Catalog record

CREATORJinchang Zhu, Jindong Li, Chengyu Zou, Rong Fu, Chao Wang, Haowei He, Menglin Yang, [0.3em] The Hong Kong University of Science, Technology (Guangzhou), [0.2em] [email protected], [0.2em] Corresponding author
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10544

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing Jinchang Zhu, Jindong Li, Chengyu Zou, Rong Fu, Chao Wang, Haowei He, Menglin Yang, [0.3em] The Hong Kong University of Science, Technology (Guangzhou), [0.2em] [email protected], [0.2em] Corresponding author Abstract Long-context adaptation is commonly treated as window scaling: increase the training sequence length, continue pretraining, and expect long-context behavior to improve. This view misses a token-level supervision mismatch : longer windows provide longer possible contexts, while causal language modeling delivers supervision one target token at a time. In packed training with document-boundary masking, each target token has its own effective left context , the visible same-document prefix at prediction time, and the resulting exposure histogram remains heavily concentrated in short and medium regimes . We introduce EXACT (Effective-context Allocation for Context Training), a supervision-allocation objective that measures this exposure distribution, assigns a fixed extra supervision weight to long effective-context targets, and distributes that extra weight by inverse frequency within the long-context tail. Across seven Qwen and LLaMA continued-pretraining (CPT) configurations, EXACT improves all twenty-eight trained/extrapolated NoLiMa and RULER comparisons in the main table. On Qwen2.5-0.5B, it improves NoLiMa by +10.09 points in trained lengths and +5.34 in extrapolated lengths, and improves RULER by +10.69 and +5.55. On LLaMA-3.2-3B, it improves RULER by +17.91 and +16.11 in trained and extrapolated regimes. Standard QA and reasoning are preserved, with a +0.24 macro change across MMLU, ARC-C, HellaSwag, WinoGrande, PIQA, and GSM8K. A distance-resolved evidence-ablation probe further shows that the gains arise in the expected regime: when relevant evidence is placed thousands of tokens away from the query, EXACT makes the model more likely to prefer the correct answer, while short-distance cases remain close to standard CPT. These results support a supervision-centric thesis: long-context adaptation depends not only on what context a model can process, but also on how strongly training supervises predictions made under long effective context. 1 Introduction Long-context language models are usually adapted by increasing the training window ( 8 ; 3 ; 19 ; 16 ) . This practice treats sequence length as the unit of long-context supervision. Sequence length is a capacity variable: it sets the maximum span a model may process in a forward pass. Supervision length is a token-level variable: it is determined by how much usable left context each supervised target actually has. A 4K or 8K packed sequence can contain many targets whose same-document left context is short, especially under document-boundary masking and realistic document length mixtures. The right unit of analysis is therefore the target token. For a token at position i , its effective left context \ell_{i} is the number of previous same-document tokens visible when predicting that token. This quantity depends on causal masking, document boundaries, packing layout, and the length distribution of the selected documents. Data selection decides which documents enter the stream and how much long text is available. The training objective decides how supervision is allocated across the tokens produced by that stream. These two layers interact, but they are distinct. Even with a long-window corpus, the training signal can remain concentrated on short and medium effective-context targets. Figure 1: Standard 16K CPT still places most token-level supervision below 4K effective context. EXACT keeps the packed stream fixed but changes per-token loss weights; when weighted objective contributions are normalized for visualization, the relative loss mass shifts into the 4K+ tail, and within that tail farther effective-context buckets receive a larger share. Figure 1 makes the mismatch visible. The packed sequence may span 16K tokens, yet each supervised target sees only the prefix of its own document segment. In the measured 16K stream, 72.6% of standard-CPT loss mass still lands below 4K effective context, leaving only 27.4% in the 4K+ tail. EXACT keeps the stream fixed and changes the supervision allocation: the 4K+ tail rises to 30.3%, and the farther buckets receive a larger share within that tail. This is the supervision-allocation bottleneck targeted by this paper: long-context adaptation pays for long windows, while a large share of training signal still lands on targets with short or medium effective context . We introduce EXACT , Effective-context Allocation for Context Training, to make this hidden allocation explicit. EXACT computes each supervised token’s effective left-context length , maps it into logarithmic buckets, and assigns fixed extra supervision weight to the long-context tail. Within that tail, the added weight is distributed by inverse bucket frequency, so rare long-effective-context targets become first-class training events rather than being washed out by the much larger population of short-context targets. The empirical pattern follows this mechanism. Across seven Qwen and LLaMA continued-pretraining configurations, EXACT improves every trained and extrapolated NoLiMa/RULER aggregate in the main table. On Qwen2.5-0.5B 4K CPT, NoLiMa rises by +10.09 points in trained lengths and +5.34 in extrapolated lengths, while RULER rises by +10.69 and +5.55. On LLaMA-3.2-3B staged 4K \rightarrow 8K CPT, RULER improves by +17.91 and +16.11 in trained and extrapolated regimes. Standard QA is preserved, with a +0.24 macro change across MMLU, ARC-C, HellaSwag, WinoGrande, PIQA, and GSM8K. The strongest normalized gains occur in regimes where long effective-context supervision should matter most. A distance-resolved evidence probe then shows the same shape inside the model: EXACT leaves short distances near standard long-window CPT parity, but opens positive context-induced margin gains from 4K to 16K. The preservation and probe results show why the long-context scores move without sacrificing ordinary QA competence: distant evidence becomes more usable. Our contributions are summarized as follows: • We identify effective-context exposure imbalance as a measurable supervision-allocation bottleneck in packed causal language modeling. • We propose EXACT , a token-level objective that assigns fixed extra supervision weight to the long-context tail and distributes it by inverse effective-context bucket frequency . • We show consistent gains on official generation-based NoLiMa and RULER evaluations across seven Qwen and LLaMA continued-pretraining configurations, with standard QA preserved and a distance-resolved evidence probe confirming stronger conversion of far context into answer likelihood. 2 Related Work Long-context capacity and positional range. A large body of work extends the amount of context that language models can process. Transformer recurrence and memory mechanisms extend usable history across segments ( 12 ; 40 ) , while sparse or approximate attention reduces long-sequence cost through structured attention patterns or kernel approximations ( 9 ; 5 ; 50 ; 26 ; 10 ) . Efficient exact-attention and KV-cache methods improve the practicality of long-window training and inference ( 13 ; 14 ; 31 ; 1 ) , and architectures such as LongNet and Infini-attention explore extremely long or unbounded contexts ( 15 ; 35 ) . A complementary line extends the positional range of pretrained models through RoPE, ALiBi, interpolation, or rescaling ( 44 ; 38 ; 7 ; 37 ; 16 ; 24 ; 47 ; 21 ) . These methods expand what context can be processed and where models can operate. EXACT studies a different training-objective question: once long windows are available, how much supervision reaches targets predicted under long same-document effective context? Long-context continued pretraining and data supply. Continued pretraining and long-context fine-tuning are standard tools for adapting LLMs to longer inputs. LongLoRA makes long-context fine-tuning efficient through sparse attention and parameter-efficient adaptation ( 8 ) .

Read the complete text