Ranked Voting based Self-Consistency of Large Language Models
Weiqin Wang, Yile Wang, Hui Huang · 2025
Read Ranked Voting based Self-Consistency of Large Language Models by Weiqin Wang, Yile Wang, Hui Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10772.txt.
Introduction
Weiqin Wang and co-authors improve model reasoning by voting over ranked answer lists in a 2025 paper. A known trick samples many reasoning paths and takes the most common final answer. Each path, though, holds other candidate answers that plain voting throws away.
Their method asks each path to rank its answers, then pools ranks across paths with election rules. Three rules are tried, from runoff style to point counts. Tests on six data sets, spanning choice and open tasks, beat plain voting for both open and closed models. Ranked ballots, they argue, squeeze more signal from the same samples.
Catalog record
| CREATOR | Weiqin Wang, Yile Wang, Hui Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-10772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ranked Voting based Self-Consistency of Large Language Models Weiqin Wang, Yile Wang, Hui Huang Abstract Majority voting is considered an effective method to enhance chain-of-thought reasoning, as it selects the answer with the highest “self-consistency” among different reasoning paths Wang et al. 2023 . However, previous chain-of-thought reasoning methods typically generate only a single answer in each trial, thereby ignoring the possibility of other potential answers. As a result, these alternative answers are often overlooked in subsequent voting processes. In this work, we propose to generate ranked answers in each reasoning process and conduct ranked voting among multiple ranked answers from different responses, thereby making the overall self-consistency more reliable. Specifically, we use three ranked voting methods: Instant-runoff voting, Borda count voting, and mean reciprocal rank voting. We validate our methods on six datasets, including three multiple-choice and three open-ended question-answering tasks, using both advanced open-source and closed-source large language models. Extensive experimental results indicate that our proposed method outperforms the baselines, showcasing the potential of leveraging the information of ranked answers and using ranked voting to improve reasoning performance. The code is available at https://github.com/szu-tera/RankedVotingSC . 1 Introduction Large language models (LLMs) have shown strong performance in recent years Ouyang et al. 2022 ; OpenAI 2023 ; Dubey et al. 2024 ; Yang et al. 2024 ; DeepSeek-AI 2024 . Chain-of-thought prompting Wei et al. 2022 further improves the performance of LLMs in commonsense Talmor et al. 2019 and mathematical Cobbe et al. 2021 reasoning tasks. Building on these advancements, Wang et al. 2023 propose a majority voting based self-consistency approach, which leverages multiple reasoning paths through sampling to identify the most self-consistent answer, thereby improving the reasoning performance of LLMs. An example of majority voting based self-consistency is shown in Figure 1 (a). In four responses by the model, option (A) was answered twice, (B) and (C) were each answered once, therefore (A) is considered the answer with the highest “self-consistency”. In each response, the model replies with only one option, thereby omitting the possibility and priority of other options, which may introduce biases in the following majority voting process. In this work, we consider obtaining ranked answers instead of only a single answer in each response and employ ranked voting based self-consistency from multiple ranked answers, which we hope can lead to more reliable self-consistency and better reasoning performance. Figure 1: Example of (a) majority voting based self-consistency among single answers Wang et al. 2023 and (b) ranked answers in four responses by models. Figure 2: Examples of the procedures for three ranked voting methods. (a) The obtained ranked answers ( k=8 ). (b) Instant-runoff voting. (c) Borda count voting. (d) Mean reciprocal rank voting. We show the example of ranked answers in Figure 1 (b). The model generates the possibility ranking of all options each time. In this case, we find that option (B) ranks in the top two positions in all four responses, while option (A) ranks last in two responses. Considering the overall results, option (A) may not be the best suitable final answer. To further decide the best answer according to the ranking information, we attempt to use ranked voting to determine the most appropriate answer based on the obtained ranked answers in responses. Ranked voting has been widely used in election systems. Specifically, we use three ranking voting methods to facilitate the final answer, including instant-runoff voting Cary 2011 , Borda count voting Emerson 2013 , and mean reciprocal rank voting. The first two methods are widely used in elections, while the latter one is related with the ranking-based MRR metric. We consider these ranked voting based approaches, compared with majority voting on single answers, can provide more reliable final answer by leveraging the ranking information of ordered answers. We validate our ranked voting based self-consistency by using advanced LLMs, including four open-source models LLaMa-3 Dubey et al. 2024 , Qwen-2.5 Yang et al. 2024 , Gemma-2 Team Gemma et al. 2024 , Phi-3 Abdin et al. 2024 , and two closed-source models GPT-3.5 OpenAI 2022 and GPT-4 OpenAI 2023 . Empirical results show that our method consistently outperforms baselines on three multiple-choice and three open-ended question-answering datasets. 2 Method We first introduce the majority voting based self-consistency baseline (§ 2.1 ) and propose obtaining the ranked answers in each LLMs’ response (§ 2.2 ). Then we provide detailed descriptions to three ranked voting methods as shown in Figure 2 , including instant-runoff voting (§ 2.3 ), Borda count voting (§ 2.4 ), and mean reciprocal rank voting (§ 2.5 ). In the final we briefly present overall comparison and how to handle tie votes (§ 2.7 ). 2.1 Majority Voting based Self-Consistency Given a model \mathcal{M} and question \mathcal{Q} , following chain-of-thought reasoning Wei et al. 2022 , we have \mathcal{R},\mathcal{A}={\mathcal{M}}(\mathcal{Q}_{1},r_{1},a_{1}...,\mathcal{Q}_{n},r_{n},a_{n},\mathcal{Q}), (1) where \{\mathcal{Q}_{j} , r_{j} , a_{j}\}_{j=1}^{n} are questions, reasoning paths, and answers in few-shot examples, respectively. \mathcal{R} and \mathcal{A} are generated reasoning path and answer of the question \mathcal{Q} . Wang et al. 2023 further propose finding the most “consistent” answer among multiple responses. In particular, by setting a high decoding temperature (e.g., \tau=0.7 ), the final answer is the majority voting results across k possible solutions \{\mathcal{R}_{i},\mathcal{A}_{i}\}_{i=1}^{k} in totally k responses: \mathcal{A}_{\text{final}}^{\text{majority}}={\rm argmax}_{\mathcal{A}}\sum_{i=1}^{k}\mathbbm{1}(\mathcal{A}_{i}=\mathcal{A}). (2) Multiple-Choice Question-Answering /* Few-Shot Examples */ Question: […] ( \mathcal{Q}_{j} ) Answer: […] ( a^{r_{1}}_{j} \succ a^{r_{2}}_{j} \succ a^{r_{3}}_{j} \succ a^{r_{4}}_{j} ) /* Test Question */ Question: George wants to warm his hands quickly by rubbing them. Which skin surface will produce the most heat? (A) dry palms. (B) wet palms. (C) palms covered with oil. (D) palms covered with lotion. ( \mathcal{Q} ) Answer: Heat is generated by friction when two surfaces rub together. The amount of friction depends on the dryness and texture of the surfaces. Dry palms have […]. Wet palms reduce […]. Palms covered with oil lower […], and palms covered with lotion reduce […]. The ranking of options by likelihood is: A > B > D > C . ( \mathcal{A}^{r_{1}} \succ \mathcal{A}^{r_{2}} \succ \mathcal{A}^{r_{3}} \succ \mathcal{A}^{r_{4}} ) Open-Ended Question-Answering /* Few-Shot Examples */ Question: […] ( \mathcal{Q}_{j} ) Answer: […] ( a^{r_{1}}_{j} \succ a^{r_{2}}_{j} \succ a^{r_{3}}_{j} \succ a^{r_{4}}_{j} \succ a^{r_{5}}_{j} ) /* Test Question */ Question: The language used in Cameroon is ( \mathcal{Q} ) Answer: Cameroon is a multilingual country located in Central Africa. However, the official languages of the country are French and English. While many indigenous languages are also spoken, they do not have the same status as the official languages. The ranking of answers by likelihood is: French > English > Pidgin English > Fulfulde > Hausa . ( \mathcal{A}^{r_{1}} \succ \mathcal{A}^{r_{2}} \succ \mathcal{A}^{r_{3}} \succ \mathcal{A}^{r_{4}} \succ \mathcal{A}^{r_{5}} ) Table 1: The examples to obtain ranked answers \mathcal{A}^{r} ( colored in blue ) on multiple-choice and open-ended question-answering tasks in few-shot settings. 2.2 From Single Answer to Ranked Answers In Eq. 1 , \mathcal{A} usually indicates a single answer such as a specific option in multiple-choice question-answering. However, it is difficult to reflect the possibility of other options from the single answer \mathcal{A} . Thus, we consider obtaining the ranked answers \mathcal{A}^{r} , which contains multiple ranked candidates according to the preference of LLMs: