Language Ranker: A Lightweight Ranking framework for LLM Decoding
Chenheng Zhang, Tianqi Du, Jizhe Zhang, Mingqing Xiao, Yifei Wang, Yisen Wang, Zhouchen Lin · 2025
Read Language Ranker: A Lightweight Ranking framework for LLM Decoding by Chenheng Zhang, Tianqi Du, Jizhe Zhang, Mingqing Xiao, Yifei Wang, Yisen Wang, Zhouchen Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21883.txt.
Introduction
Chenheng Zhang and six co-authors, Tianqi Du, Jizhe Zhang, Mingqing Xiao, Yifei Wang, Yisen Wang. Zhouchen Lin, look at the last step of language model generation, where scores become the final answer. Reward models can rerank candidate answers but are heavy and costly, and ordinary decoding leaves value on the table.
Borrowing the ranking stage from recommender systems, the team adds a tiny module that reorders candidate responses using features the base model already computed. The add-on holds under half a million parameters, yet across a wide range of tasks it matched the quality of far larger reward models. The work reframes decoding as a ranking problem and suggests that careful selection, not bigger scoring models, may be the cheaper path to better answers.
Catalog record
| CREATOR | Chenheng Zhang, Tianqi Du, Jizhe Zhang, Mingqing Xiao, Yifei Wang, Yisen Wang, Zhouchen Lin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21883
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language Ranker: A Lightweight Ranking framework for LLM Decoding Chenheng Zhang, Tianqi Du, Jizhe Zhang, Mingqing Xiao, Yifei Wang, Yisen Wang, Zhouchen Lin Abstract Conventional research on large language models (LLMs) has primarily focused on refining output distributions, while paying less attention to the decoding process that transforms these distributions into final responses. Recent advances, such as scaling the computation of inference time with reward models, have underscored the importance of decoding, but these methods often suffer from high computational costs and limited applicability. In this paper, we revisit LLM generation through the lens of recommender systems , conceptualizing the decoding process as analogous to the ranking stage in recommendation pipelines. From this perspective, we observe that both traditional decoding methods and reward models exhibit clear limitations such as redundancy. Motivated by this insight, we propose Language Ranker , a novel framework that introduces a lightweight module to rerank candidate responses using features extracted by the base model. Experiments across a wide range of tasks show that Language Ranker achieves performance comparable to large-scale reward models, while requiring only <0.5M additional parameters, significantly reducing the computational overhead during both training and inference stages. This highlights the efficiency and effectiveness of our method, showcasing its potential to fully unlock the capabilities of LLMs. The implementation is released at https://github.com/chenhengzh/language_ranker . 1 Introduction Traditional research on enhancing the capabilities of large language models (LLMs) has primarily focused on improving the quality of output distributions through approaches such as scaling up model sizes ( kojimaLargeLanguageModels2023 ) , fine-tuning for specific tasks (SFT) ( gaoParameterEfficientFineTuningDiscrete2024 ; malladiFineTuningLanguageModels2023 ) , and reinforcement learning with human feedback (RLHF) ( rafailovDirectPreferenceOptimizationb ; ouyangTrainingLanguageModels ) . However, the decoding process, which converts the output distributions into final responses, has not received sufficient attention. Current decoding strategies, including top- k sampling ( fan-etal-2018-hierarchical ; holtzman-etal-2018-learning ) , self-consistency ( wangSelfConsistencyImprovesChain2023 ) , and contrastive decoding ( liContrastiveDecodingOpenended2023 ) , are largely rule-based and task-specific, limiting their ability to fully exploit the potential of LLMs’ powerful output distributions. It is found that if an oracle could select the best response from multiple samples generated by a model, the performance of a 7B model could even surpass that of a 70B model as the sample number increases ( brownLargeLanguageMonkeys2024 ) . This finding highlights the tremendous potential of the decoding process in maximizing model performance. To approximate the oracle, recent studies on inference-time computing ( snellScalingLLMTestTime2024a ; wuInferenceScalingLaws2024 ; setlurRewardingProgressScaling2024 ) have introduced reward models to select the best response. While these methods demonstrate strong performance across various tasks, the reliance on auxiliary reward models significantly increases computational and time overhead during both training and inference, thereby limiting their scalability and applicability in broader contexts. Figure 1 : The comparison among existing methods, recommender system and our Language Ranker. The two charts on the left highlight the limitations of existing decoding strategies and reward models, in contrast to the recommender system pipeline shown in the top-right. Language Ranker addresses these limitations by incorporating a feature-shared, learnable, and lightweight ranker. Table 1 : Comparison of parameters and performance between the Language Ranker and reward models, using Llama3.1-8B-Instruct as the base model. Method Training Stage Inference Stage Performance Trainable GPU-Loaded Sampling Ranking Language Ranker <0.5M <0.5M 8.2B <0.5M ★★★ RM (llama8B-LoRA) 176M 8.2B 8.2B 8.4B ★★★ RM (gpt2) 137M 137M 8.2B 137M ★✩✩ To address these limitations, we rethink LLMs through the lens of recommender systems. As shown in Figure 1 , each LLM can be viewed as a special recommender system , where the input serves as the user information, and the model’s role is to recommend the most appropriate response as the “item” tailored to the user’s needs. Therefore, the model backbone, language head, and decoding process correspond directly to the feature engineering, retriever, and ranker in a traditional recommender system ( zhangDeepLearningBased2020 ) . When a user provides an input, the model backbone first extracts user features, represented as the hidden states of the last token. The language head then generates a coarse response distribution. Finally, a predefined decoding strategy samples several candidate responses from this distribution and selects the most suitable one among them. In this analogy, the limitations of both existing decoding strategies and reward models become evident. As illustrated in Figure 1 , current decoding strategies are typically simple and rule-based, neglecting the crucial role of reranking model responses. Meanwhile, reward models, though effective as rankers, introduce substantial computational overhead both in training and inference. From the perspective of recommender systems, these methods essentially redo the feature engineering for ranking from scratch, ignoring the features already extracted during the recall stage that could have been shared. This redundancy leads to significant unnecessary computations and inefficiency. In this paper, we propose Language Ranker , inspired by recommender systems, to address aforementioned shortcomings of existing methods. The Language Ranker incorporates a carefully designed lightweight ranker to rerank candidate responses generated by the base model, which constitutes the only trainable component in our framework. As illustrated in Figure 3 , the earlier layers of the base model can be viewed as shared feature engineering for both the retriever and ranker, similar to recommender systems. Once the candidate responses are sampled, the ranker utilizes the extracted features to rerank the candidates and identify the most appropriate response. As shown in Table 1 , by leveraging the representations of base models, our method achieves performance comparable to that of the Llama8B-based reward model, while requiring only <0.5M additional parameters, significantly reducing the computational overhead during both training and inference stages. Table 4 shows that the lightweight ranker supports both training and inference independently on CPUs, demonstrating the potential to build a personalized Language Ranker. As illustrated in Figure 3 , the base model can be paired with different rankers to enhance capabilities across various dimensions. The base model is expected to run on high-resource central nodes, while the rankers can be deployed on edge nodes or even on users’ local devices, allowing continual learning and personalized adaptation to diverse user needs. In conclusion, the main contributions of this paper are: • We reinterpret LLMs through the lens of recommender systems, revealing the limitations of existing decoding strategies and reward models while highlighting the potential of the decoding process. • We propose Language Ranker, a novel and lightweight ranking framework for LLMs that is both efficient and effective. It allows a single base model to be flexibly paired with different rankers, allowing personalized adaptation to diverse user needs. • We conduct extensive experiments across diverse tasks and multiple base models, demonstrating that our framework achieves performance comparable to large-scale reward models while requiring <0.5M additional parameters, thereby substantially reducing computational overhead during both training and inference. Figure 2 : The Framework of Language Ranker. The base model first generates multiple candidate responses. The hidden states corresponding to the final tokens of both the instruction and each candidate response are then extracted from a predefined layer and used as features. Finally, the ranker selects the most appropriate response based on these features. Figure 3 : Personalized Language Ranker. We can pair a single base model with different rankers to enable personalized adaptation for diverse