From Projection to Prediction: Beyond Logits for Scalable Language Models
Jianbing Dong, Jianbin Chang · 2025
Read From Projection to Prediction: Beyond Logits for Scalable Language Models by Jianbing Dong, Jianbin Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17599.txt.
Introduction
Jianbing Dong and Jianbin Chang propose a leaner way to train large language models at the final output step. Standard training first projects the model's inner states into scores for every word in its vocabulary, then compares those scores with the target word. That score table is huge, scaling with batch, length, and vocabulary, and it must sit in graphics memory even though only one target word per spot is ever used.
Their method folds projection and loss into a single operation, computing the loss straight from the inner states and the target words. Skipping the full score table cuts memory use and data movement, which should lift training speed and allow larger runs. The paper targets a quiet bottleneck that grows worse as vocabularies and models expand.
Catalog record
| CREATOR | Jianbing Dong, Jianbin Chang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-17599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Projection to Prediction: Beyond Logits for Scalable Language Models Jianbing Dong, Jianbin Chang Abstract Training Large Language Models (LLMs) typically involves a two-stage pipeline at the output layer: hidden states are projected into vocabulary logits via a linear transformation (lm_head), followed by cross-entropy loss computation against target tokens. While conceptually simple, this design incurs substantial overhead. The intermediate logits tensor, with dimensions proportional to batch size, sequence length, and vocabulary size, must be fully materialized in GPU memory, even though only one target token per position is ultimately used. This leads to significant memory footprint and bandwidth comsumption, limiting scalability and slowing training throughput. In this work, we introduce a novel approach to integrates the output projection and loss prediction into a single operation. By directly computing the loss from hidden states and target tokens, our approach bypasses explicit logits materialization. This design reduces memory usage and alleviates bandwidth pressure. Experiments on LLM training demonstrate that our method achieves substantial memory savings and measurable speedups compared to the standard two-stage pipeline, enabling large batch sizes and longer sequences without sacrificing accuracy. Our work highlights the benefits of rethinking the boundary between projection and prediction, offering a practical systems optimization for efficient LLM training. Keywords: LLM, Output Projection, Cross-Entropy Loss, Memory Footprint Reduction 1 Introduction L arge language models (LLMs) have rapidly advanced the state of natural language processing, powering applications in dialogue systems [ 21 , 7 ] , code generation [ 2 ] , and scientific discovery [ 23 ] . Their success has been enabled by scaling both model size and training data, but this scaling comes at a steep computational and memory cost [ 5 , 3 , 1 ] . As models grow to billions of parameters and vocabularies reach hundreds of thousands of tokens, even seemingly routine components of the training pipeline can become critical bottlenecks [ 17 ] . A particularly important bottleneck lies in the output layer. In every standard LLM training workflow, hidden states are projected into vocabulary logits through a large linear transformation known as the lm_head . These logits are then consumed by a cross-entropy loss function to predict target tokens. This two-stage design is not an implementation detail but a canonical requirement: the lm_head projection and cross-entropy loss together define how probability distributions over vocabularies are learned [ 22 ] . However, prevailing implementations invariably insist on materializing the entire logits tensor in GPU memory [ 6 ] , a practice that inflates computational overhead and saturates bandwidth despite its limited utility. The tensor’s dimensions scale with batch size, sequence length, and vocabulary size, leading to substantial memory footprint and bandwidth consumption. Crucially, only one target logit per position is ultimately used in the numerator of the loss, meaning the vast majority of the computed logits are discarded. This mismatch between computation and utility limits scalability and slows training throughput. Recent research has explored kernel fusion and operator optimization in other parts of the LLM training stack, such as online softmax [ 8 ] , attention mechanisms [ 18 , 19 ] , and optimizer updates [ 13 , 14 ] . These efforts highlight the importance of reducing redundant memory movement and improving arithmetic intensity. To the best of our knowledge, the output projection and loss computation — despite being a mandatory stage in every LLM pipeline — have remained largely untouched [ 22 ] . Existing frameworks continue to treat the lm_head and cross-entropy loss as separate modules, leaving untapped opportunities for optimization. In this work, we propose a fused kernel implementation that integrates the output projection and loss prediction into a single operation. By directly computing the loss from hidden states and target tokens, our approach bypasses explicit logits materialization while preserving the standard training semantics. This design reduces memory usage, alleviates bandwidth pressure, and improves throughput. Experiments on LLM training demonstrate that our method achieves substantial memory savings and measurable speedups compared to the canonical two-stage pipeline, enabling larger batch sizes and longer sequences without sacrificing accuracy. Our contributions underscore the value of rethinking the boundary between projection and prediction. By collapsing these stages into a unified kernel, we provide a practical systems optimization that addresses a critical bottleneck in the universally adopted LLM training workflow. Beyond immediate performance gains, this work contributes to the broader effort of making large-scale language modeling more efficient, sustainable, and accessible. 2 Related Work Optimizing the efficiency of large language model training has been a central focus of recent systems research. A recurring theme is the reduction of redundant memory movement and the fusion of operations to improve arithmetic intensity [ 15 ] . These efforts demonstrate that even small changes to core primitives can yield substantial performance gains at scale. 2.1 Softmax Optimizations Online Softmax [ 8 ] introduced a memory-efficient reformulation of the softmax operator that reduces redundant memory accesses by computing the normalization term in a single pass over the input vector. This approach improves GPU performance for bandwidth-bound workloads, achieving up to 1.3 \times speedup compared to the standard safe softmax implementation. Furthermore, by fusing Online Softmax with the commonly used Top-K operation in beam search, they demonstrated up to 5 \times performance gains. This work highlights how kernel-level optimizations in seemingly routine components of the training stack can deliver substantial efficiency improvements, influencing subsequent research on operator fusion and memory-aware kernel design in LLM systems. 2.2 Attention Mechanism Optimizations The attention mechanism has been another major target for kernel fusion. FlashAttention [ 18 , 19 ] introduced a highly optimized, IO-aware algorithm that computes attention in a tiled fashion, thereby avoiding the materialization of large intermediate matrices. By tightly coordinating memory access with computation, FlashAttention achieves significant speedups and memory savings, enabling training with longer sequence lengths and larger batch sizes. This line of work illustrates how re-engineering core components of the transformer architecture [ 20 ] can unlock substantial efficiency gains and expand the practical limits of large-scale model training. 2.3 Operator Fusion in LLM Systems Operator fusion has also been widely adopted in industry-grade libraries. NVIDIA’s TransformerEngine [ 11 ] provides a suite of fused CUDA kernels tailored for transformer workloads [ 20 ] , including GEMM combined with bias addition and activation functions, fused layer normalization, and optimized softmax variants. By collapsing multiple sequential operations into a single kernel, TransformerEngine eliminates redundant memory movement and reduces kernel launch overhead, thereby improving arithmetic intensity and throughput. These fused primitives are designed to exploit Tensor Cores and mixed-precision formats such as FP8 and BF16, enabling efficient training and inference of very large language models. TransformerEngine exemplifies how production systems integrate operator fusion into the transformer stack, complementing research efforts like Online Softmax and FlashAttention, and reinforcing the broader trend of re-engineering fundamental components for efficiency. Together, these efforts highlight a broader trajectory: efficiency gains in LLM training often arise from collapsing boundaries between traditionally separate operations. In line with this trajectory, we introduce a kernel-level optimization that pushes forward the broader agenda of reducing memory movement and improving arithmetic intensity in LLM systems. 3 Methodology 3.1 Standard Output Projection and Loss Prediction In conventional LLM training, the output layer consists of two