Majority Voting for Code Generation

Tim Launer, Jonas Hübotter, Marco Bagatella, Ido Hakimi, Andreas Krause · 2026

Read Majority Voting for Code Generation by Tim Launer, Jonas Hübotter, Marco Bagatella, Ido Hakimi, Andreas Krause (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15618.txt.

Introduction

Launer and four colleagues study functional majority voting for code generation, which picks one answer from many model tries by how each try runs on test inputs. The test-time trick lifts LiveCodeBench scores a lot without much extra compute. Used to combine label-free test-time reinforcement learning, it raises pass-at-1 on holdout tasks but shows no self-gains beyond the base model's ceiling.

Catalog record

CREATORTim Launer, Jonas Hübotter, Marco Bagatella, Ido Hakimi, Andreas Krause
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15706
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15618

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Majority Voting for Code Generation Tim Launer, Jonas Hübotter, Marco Bagatella, Ido Hakimi, Andreas Krause Abstract We investigate Functional Majority Voting (FMV), a method based on functional consensus for code generation with Large Language Models, which identifies a representative solution from multiple generations using their runtime execution signatures on test inputs. We find that FMV is an effective test-time inference strategy, substantially boosting performance on LiveCodeBench without a large compute overhead. Furthermore, we extend the utility of functional consensus and apply it as an aggregation strategy for label-free Test-Time Reinforcement Learning. We demonstrate that this increases pass@1 on holdout tasks, but find no evidence of self-improvement beyond the base model’s performance ceiling. 1 Introduction A central challenge in inference-time scaling of Large Language Models (LLMs) for improved reasoning is discerning promising generations from hallucinations. While models can reliably generate plausible-looking chains of thought, their reasoning capabilities often remain brittle ( Nezhurina et al. 2024 ; Berglund et al. 2024 ) . In the absence of an external oracle verifier, recent approaches leverage strategies such as consensus sampling ( Wang et al. 2023 ; Samadi et al. 2025 ; Jiang et al. 2026 ) or test-time reinforcement learning ( Zuo et al. 2025 ) to improve performance at test time. However, applying these methods to code generation is a non-trivial challenge ( Li et al. 2022 ; Li et al. 2025 ) . Yet, code generation has a unique property: executability . By executing generated programs against self-generated inputs, we can construct a “functional consensus” that serves as a proxy for ground truth ( Shi et al. 2022 ; Chen et al. 2024 ; Menet et al. 2026 ) . To this end, we investigate Functional Majority Voting (FMV) . While execution-based consensus has previously been employed for decoding ( Shi et al. 2022 ) or as a component in recursive decomposition ( Chen et al. 2024 ) , we isolate the soft functional voting mechanism itself as FMV to study its standalone scaling properties and its potential as a training signal. We demonstrate that this execution based signal serves a dual purpose: • Standalone functional consensus for test-time inference: We demonstrate that soft functional consensus significantly boosts performance over the baseline. • FMV-based Test-Time Reinforcement Learning: We extend the utility of FMV beyond inference, using it as a pseudo-label generator for Test-Time Reinforcement Learning (TTRL) ( Zuo et al. 2025 ) in two flavors. As a reward signal, we use the consensus execution vectors to verify rollouts (matching the consensus yields reward 1.0, otherwise 0.0). 2 Related Work Test-Time Inference. The foundation of our test-time inference strategy introduced in section 3 is the “self-consistency” decoding method introduced by Wang et al. 2023 , who demonstrated that for complex reasoning tasks such as math, sampling a diverse set of reasoning paths and marginalizing over final answers significantly outperforms base performance. The intuition is that while incorrect reasoning paths are stochastic and diverse, correct reasoning paths tend to be consistent, leading to the same final answer. Applying this consistency principle to code generation requires moving beyond text matching. While approaches like CodeT ( Chen et al. 2023 ) rely on generating full input-output pairs, we simplify this by following Shi et al. 2022 , requiring only generated inputs to establish functional consensus. Furthermore, Chen et al. 2024 introduced functional consensus as a sub-component within FunCoder , a recursive divide-and-conquer planning framework. While effective, their approach requires complex recursive problem decomposition. In contrast, we isolate the functional voting mechanism to study its scaling properties, demonstrating that execution consensus alone provides significant gains in both training and inference. Figure 1: Candidate solution programs c_{1},\dots,c_{5} are sampled from a Large Language Model and evaluated against test inputs. Invalid candidates (e.g., c_{2} ) are discarded. The consensus c^{*} (here c_{1} ) is selected by maximizing the Score S(c_{i}) . Another execution-based clustering approach was recently employed by Samadi et al. 2025 in GenCluster to achieve gold-medal performance on International Olympiad of Informatics (IOI) tasks. Similar to our method, they used execution based methods for finding promising candidates, though contrary to our approach they select final solutions using re-ranking (round-robin style) tournaments. Finally, Jiang et al. 2026 introduce semantic voting, a method that relaxes hard matching consensus methods and instead uses soft matching based on semantic similarity. Specifically, each generated response is encoded into a vector using a sentence embedding model, and its voting score is computed as the average cosine similarity with all other responses. Test Time Reinforcement Learning. Our investigation in Section 4.2 is mainly inspired by Zuo et al. 2025 , who introduced Test-Time Reinforcement Learning (TTRL), a method for training a pre-trained model at test-time using RL without ground-truth labels. To construct a reward signal, TTRL generates multiple candidate outputs from the model and identifies a consensus output via self-consistency, which is used as the ground-truth label for training. They demonstrate that for mathematical reasoning, a model trained with TTRL can solve problems far exceeding its initial baseline without any human supervision. 3 Method: Functional Majority Voting Let \mathcal{M} be a model that generates N candidate programs \mathcal{C}=\{c_{1},\dots,c_{N}\} given a prompt P . We assume access to a set of K test inputs 1 1 1 If not available off the shelf, test case inputs can reliably be sampled from an LLM. \mathcal{X}=\{x_{1},\dots,x_{K}\} , but not their ground-truth outputs. We execute every candidate c_{i} on every input x_{k} and collect the resulting output strings o_{i,k} to obtain an execution vector \mathbf{o}_{i}=[o_{i,1},\dots,o_{i,K}] . We define the set of valid candidates \mathcal{C}_{val} as those that execute without runtime errors, invalid formats, or time outs on all inputs. The goal is to extract a consensus program c^{*}\in\mathcal{C}_{val} , representing the majority vote. FMV (The Functional Medoid 2 2 2 A “medoid” is the representative program with minimal disagreement to all other programs. ). Paralleling Shi et al. 2022 and Chen et al. 2024 , we adopt a functional consensus scoring mechanism to select the most robust single consensus program c^{*} for inference, and apply it directly to the candidate programs. Specifically, we identify the candidate that maximizes pairwise functional agreement with the ensemble of execution vectors. To this end, we define the FMV score S(c_{i}) for each c_{i}\in\mathcal{C}_{val} : S(c_{i})=\sum_{j\neq i}\sum_{k=1}^{K}\mathbb{I}(o_{i,k}=o_{j,k}) (1) The consensus program is then immediately given by c^{*}=\mathrm{argmax}_{c_{i}}S(c_{i}) (see Figure 1 ). Intuitively, S(c_{i}) computes the average agreement across test cases between the output of the i -th candidate and those of other candidates. Standard consensus methods often require strict equivalence, meaning two programs must have identical output vectors to be grouped. This is brittle; if valid solutions differ on even a single edge case, strict voting fails to find a majority. To resolve this, this score uses a soft similarity metric that awards “partial credit” for matching on subsets of test cases. Pointwise-FMV (Synthetic Targets). For test-time training with TTRL ( Zuo et al. 2025 ) , we can relax the requirement of selecting a single consensus candidate program. Therefore, as an alternative to using the execution vector to obtain the consensus program c^{*} from above, we can instead construct a purely synthetic target execution vector \mathbf{y}^{*} : For each test case

Read the complete text