ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learning Chu Zhao, Enneng Yang, Yuting Liu, Jianzhe Zhao, Guibing Guo Abstract Test-time reinforcement learning generates multiple candidate answers via repeated rollouts and performs online updates using pseudo-labels constructed by majority voting. To reduce overhead and improve exploration, prior work introduces tree-structured rollouts, which share reasoning prefixes and branch at key nodes to improve sampling efficiency. However, this paradigm still faces two challenges: (1) high-entropy branching can trigger rollout collapse, where the branching budget concentrates on a few trajectories with consecutive high-entropy segments, rapidly reducing the number of effective branches; (2) early pseudo-labels are noisy and biased, which can induce self-reinforcing overfitting, causing the policy to sharpen prematurely and suppress exploration. To address these issues, we propose Entropy–Confidence Hybrid Group Relative Policy Optimization (ECHO). During rollout, ECHO jointly leverages local entropy and group-level confidence to adaptively control branch width, and further introduces online confidence-based pruning to terminate persistently low-confidence branches, avoiding high-entropy traps and mitigating collapse. During policy updates, ECHO employs confidence-adaptive clipping and an entropy–confidence hybrid advantage shaping approach to enhance training robustness and mitigate early-stage bias. Experiments demonstrate that ECHO achieves consistent gains on multiple mathematical and visual reasoning benchmarks, and generalizes more effectively under a limited rollout budget. The implementation code is available at https://github.com/user683/ECHO † † affiliationnotice: Equal contribution 1 Introduction Current language models ( 16 ; 1 ; 30 ) often rely on high-quality human-labeled data for preference alignment, yet in many scenarios such labels are difficult to obtain or prohibitively expensive. To address this, researchers have proposed Test-Time Reinforcement Learning (TTRL) ( 39 ) , which improves reasoning capability by optimizing on self-feedback signals generated during inference, without requiring external supervision. In a typical TTRL setup, the model produces multiple candidate solutions for the same query, constructs pseudo-labels via self-consistency ( 36 ) or majority voting, and performs online updates accordingly. However, this paradigm is highly sensitive to the rollout budget: to obtain sufficiently stable pseudo-labels, conventional chain-based rollouts often require dozens to hundreds of samples, incurring substantial parallel sampling costs. To reduce this overhead and improve exploration, recent works ( 17 ) introduce tree-search-based rollouts ( 12 ; 9 ) , which share reasoning prefixes and branch at key decision points to achieve broader and more efficient sampling coverage under the same token budget, thereby producing more reliable learning signals at lower cost. Despite the effectiveness of the above approaches, existing TTRL methods still face two key challenges in practice. First, high-entropy branching can trigger rollout collapse . ETMR ( 17 ) branches at high-entropy nodes, but under majority-vote pseudo-labeling, the branching budget is repeatedly consumed by a small number of trajectories with consecutive high-entropy steps. As a result, the number of effective branches quickly shrinks, the search tree degenerates into an almost chain-like rollout, and pseudo-labels become dominated by homogenized trajectories ultimately reducing exploration coverage and degrading the quality of learning signals. Second, early pseudo-label bias leads to self-reinforcing overfitting . In the early stage of training, pseudo-rewards are noisy and biased, which can push the policy to prematurely converge to a locally “high-scoring” solution and form a feedback loop: the output distribution rapidly sharpens (entropy drops), exploration collapses, and both later-stage performance and generalization are constrained. Following the existing work ( 5 ) , we conduct two types of statistics over ETMR rollouts: (i) high-entropy continuity, which measures the proportion of consecutive high-entropy segments and their length distribution within each trajectory, to characterize whether high-entropy steps persistently cluster along local paths; and (ii) branch budget allocation, which summarizes how the fixed branching budget is distributed across trajectories (e.g., the number of effective branches per batch and the Top-3 budget share), to quantify budget concentration and identify rollout collapse. The results in Figure 2 further validate this phenomenon: ETMR’s branching budget is heavily skewed toward a few trajectories, so most rollouts effectively cover only 1–3 branches, exhibiting clear branching imbalance. The right panel shows that the majority of the budget is dominated by the top-ranked trajectories, while the remaining candidates receive almost no exploration resources, thereby weakening sampling coverage and reducing pseudo-label diversity. Figure 1 : Performance overview of ECHO algorithm. To address the above challenges, we propose Entropy and Confidence Hybrid Group Relative Policy Optimization (ECHO). For Challenge 1, we introduce an entropy-confidence hybrid tree search. During node expansion, ECHO jointly models local entropy and group-level confidence, and dynamically balances them with adaptive weights. It shrinks the branching width in high-confidence regions to suppress unproductive expansions, while increasing the branching width in high-entropy and low-confidence regions to strengthen exploration and improve diversity. To prevent the search from being persistently trapped in highly uncertain high-entropy traps, we further incorporate online confidence-based pruning. Specifically, we use the minimum confidence within a sliding window along each trajectory as a quality criterion, and immediately terminate and prune a branch when it falls below an adaptive threshold estimated by warm-up quantiles. This mechanism distinguishes reasonable transient exploration from persistently low-confidence erroneous reasoning chains, thereby mitigating high-entropy collapse and improving both reasoning efficiency and trajectory quality. For Challenge 2, ECHO applies confidence-adaptive clipping to couple the update magnitude with sample reliability, preventing noisy pseudo-labels from inducing overly aggressive updates. In addition, we introduce entropy and confidence joint advantage shaping, which increases the learning weight on tokens from effective but low-confidence trajectories. This design enhances training robustness and alleviates early-stage bias-induced overfitting and entropy collapse. As shown in Figure 1 , ECHO consistently outperforms strong baselines on representative reasoning benchmarks, demonstrating robust gains on both mathematical and general QA tasks. The radar plot further confirms that these improvements are broad and balanced across all evaluated datasets, rather than concentrated on a single benchmark. Our contributions are summarized as follows: (1) We propose Entropy–Confidence Hybrid Group Relative Policy Optimization, which jointly regulates local entropy and group-level confidence during rollout via adaptive weighting, and incorporates online pruning based on sliding-window confidence and warm-up quantile thresholds to suppress persistently low-confidence branches and alleviate rollout collapse. (2) We introduce the ECHO update mechanism, which combines confidence-adaptive clipping with entropy–confidence joint advantage shaping to reduce the impact of noisy pseudo-labels and to strengthen learning from tokens on effective yet low-confidence trajectories, thereby mitigating early-stage bias and entropy collapse while improving robustness and generalization. (3) Extensive experiments on mathematical and visual reasoning benchmarks demonstrate the method’s consistent gains. Figure 2 : Empirical study of high-entropy rollout collapse in entropy-driven tree search for TTRL. 2 Preliminaries Recent studies ( 35 ; 17 ; 6 ) have moved away from external annotations and instead exploit model-intrinsic statistics from token probability distributions to score, filter, and guide reasoning trajectories. Two widely used signals are: (1) Token Entropy. Given the predicted distribution P_{i} at position i , token entropy is H_{i}=-\sum_{j}P_{i}(j)\log P_{i}(j), (1) where P_{i}(j) denotes the probability of the j -th vocabulary token. Lower H_{i} indicates a sharper (more certain) distribution. (2) Token Confidence Score. Let \mathrm{Top}k(P_{i}) denote the set of top- k tokens ranked by P_{i}(\cdot) . We define a token-level confidence score as the mean probability mass over these candidates: C_{i}=\frac{1}{k}\sum_{v\in\mathrm{Top}k(P_{i})}P_{i}(v), (2) where k is a fixed hyperparameter. Larger C_{i} indicates higher token-level confidence. Building on these signals, prior work ( 17 ) adopts tree-structured rollouts in test-time RL, typically branching at high-entropy positions to improve diversity under a limited budget. However, high entropy-only branching often suffers from high-entropy collapse , where rollout budget over-concentrates on persistently uncertain regions, resulting in imbalanced exploration and degraded trajectory quality. We address this by an entropy–confidence jointly optimized tree-search framework that co-regulates uncertainty and confidence for branch triggering and budget allocation. Figure 3 : Overview of our proposed framework. Given a query, the LLM/VLM performs entropy–confidence guided tree-search rollouts with adaptive branching and confidence-based online pruning. The surviving trajectories produce candidate answers y_{i} , which are aggregated by majority voting to obtain y^{\prime} . We then compute hybrid shaped advantages A_{i}^{\mathrm{hyb}} for policy optimization, while pruning branches whose grouped confidence falls below \tau_{\text{prune}} to save budget and prevent high-entropy collapse. 3 Methodology This section details ECHO. Section 3.1 presents an entropy–confidence guided tree-structured rollout, Section 3.2 introduces confidence-adaptive clipping for stable updates under noisy pseudo-rewards, and Section 3.3 proposes entropy–confidence hybrid advantage shaping to emphasize uncertain yet decision-critical tokens. Figure 3 overviews the framework. 3.1 Tree-Structured Rollout with Entropy–Confidence Hybrid Optimization In this section, we introduce an entropy–confidence guided tree-structured rollout. It uses token-level entropy and confidence, with window-based smoothing, to adaptively decide where to branch and how many branches to allocate. Online pruning further terminates degenerate branches early, alleviating high-entropy collapse and improving rollout efficiency under a limited inference budget. Entropy Increment and Window Smoothing. Existing work ( 6 ) leverages token-level confidence to prune chain-based rollouts: when confidence stays low or shows a clear degradation trend, generation is terminated early to avoid wasting tokens and to improve estimation efficiency. Inspired by this work, we jointly model token entropy and token-level confidence , and use window-based smoothing to obtain stable signals for deciding when to branch and how wide to branch in tree-structured rollouts. Concretely, to avoid single-step noise, we apply temporal smoothing to both confidence and entropy. Let W_{G} , W_{T} , and W_{H} denote the group, tail, and entropy window sizes, respectively. To keep the statistics well-defined in the early steps, we use effective window lengths W_{G}(t)=\min(W_{G},t) , W_{T}(t)=\min(W_{T},t) , and W_{H}(t)=\min(W_{H},t-1) . We define grouped confidence as the moving average over the most recent W_{G}(t) steps: C_{t}^{G}=\frac{1}{W_{G}(t)}\sum_{s=t-W_{G}(t)+1}^{t}C_{s}. (3) Similarly, tail confidence is defined as C_{t}^{\text{tail}}=\frac{1}{W_{T}(t)}\sum_{s=t-W_{T}(t)+1}^{t}C_{s}. (4) For entropy, we compute the historical mean entropy over the past W_{H}(t) steps: \bar{H}_{t-1}=\frac{1}{W_{H}(t)}\sum_{s=t-W_{H}(t)}^{t-1}H_{s}. (5) \Delta H_{t}\triangleq\bar{H}_{t}-\bar{H}_{t-1} is the entropy increment at step t . Branch Width: Entropy–Confidence Joint Scheduling. Once a branching point is triggered, we determine the branch width B_{t} by jointly considering (i) the local token-distribution entropy and (ii) the grouped confidence across rollouts. At decoding step t for a given prompt x , we define the predictive entropy at the current node as H_{t}:=\mathcal{H}\!\big(\pi_{\theta}(\cdot\mid x,y_{\delta_{\text{upper}},\\ 0,&\text{otherwise}.\end{cases} (9) If r_{t}\geq S_{\Delta} , we prune the branch (entropy-spike event), where \delta_{\text{upper}} is the spike threshold and S_{\Delta} is the required consecutive length. Overall, these three criteria provide complementary early-stopping signals: low-confidence pruning removes branches stuck in globally unreliable regions, tail-decline pruning captures trajectories with steadily deteriorating confidence, and entropy-spike pruning filters out branches entering persistently unstable uncertainty regimes. Together, online pruning reallocates rollout budget toward higher-quality trajectories and reduces the risk of high-entropy traps. Majority Voting Reward Function. Following existing TTRL methods ( 39 ; 17 ) , we also employ voting to generate pseudo-labels. For each prompt x , we sample G candidate responses \{o_{i}\}_{i=1}^{G} under the policy, where o_{i} denotes the i -th response. Let \mathrm{answer}(o_{i}) be the final answer extracted from o_{i} and \mathbb{I}\{\cdot\} be the indicator function. We select the majority-voted prediction as the estimated label: \hat{y}=\arg\max_{a}\sum_{i=1}^{G}\mathbb{I}\big(\mathrm{answer}(o_{i})=a\big). (10) Using \hat{y} , we assign a rule-based reward to each candidate trajectory: R_{i}\;=\;\mathbb{I}\big(\mathrm{answer}(o_{i})=\hat{y}\big)\in\{0,1\}. (11) 3.2 Confidence-Adaptive Clipping TTRL typically relies on pseudo-rewards that are noisy and biased at the early stage. This early estimation bias can repeatedly push the policy toward apparently high-reward neighbors favored by the current sampling distribution, even when these neighbors are not truly reliable. Under a fixed clipping radius, such spurious high-reward trajectories may dominate the update, causing rapid distribution sharpening (entropy drop), premature exploitation, and consequently overfitting with exhausted exploration. To mitigate this issue, we make the trust region adaptive to confidence , so that unreliable early signals are prevented from inducing overly aggressive updates. We define the per-token importance ratio at decoding step t as r_{i,t}(\theta)=\frac{\pi_{\theta}\!\left(y_{i,t}\mid x,y_{i,0 is a small constant for numerical stability.We broadcast this trajectory-level advantage to each token on the same trajectory, i.e., A^{\mathrm{grp}}_{g,i,t}\equiv A^{\mathrm{grp}}_{g,i} for all t\in\{1,\dots,|o_{i}|\} . We then introduce an entropy–confidence hybrid shaping signal that emphasizes learning on uncertain regions: S_{i,t}=\alpha\,H_{i,t}+\beta\,{(1-C}_{i,t}), (16) where H_{i,t} is the token entropy at step t on trajectory o_{i} , C_{i,t}\in(0,1] is the token-level top- k confidence computed from the logits, and H_{i,t} and (1-C_{i,t}) denote masked-whitened versions of entropy and inverse confidence, respectively. The coefficients \alpha,\beta weight the two components. Finally, we incorporate S_{i,t} into the token-level advantage via A^{\mathrm{hyb}}_{i,t}=A^{\mathrm{grp}}_{g,i}\bigl(1+aS_{i,t}\bigr), (17) where a>0 is a scaling factor. This hybrid shaping biases policy updates toward informative yet uncertain decisions, complementing confidence-adaptive clipping to reduce early-stage overfitting and maintain exploration. 3.4 Overall Objective Combining confidence-adaptive clipping and entropy–confidence hybrid advantages, we obtain the ECHO objective: \begin{split}&\mathcal{L}_{\text{ECHO}}(\theta)=\mathbb{E}_{x,\{o_{i}\}}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\Big(\min\Big(r_{i,t}(\theta)\,A^{\mathrm{hyb}}_{i,t},\;\\ &\mathrm{clip}\!\big(r_{i,t}(\theta),\,1-\epsilon(o_{i}),\,1+\epsilon(o_{i})\big)\,A^{\mathrm{hyb}}_{i,t}\Big)\\ &-\beta_{\text{KL}}\,D_{\text{KL}}\!\Big(\pi_{\theta}(\cdot\mid x,y_{i,0,\ r>1+\epsilon,\\ (1-\epsilon)A,&A<0,\ r<1-\epsilon,\\ rA,&\text{otherwise}.\end{cases} (24) Proof. The statement follows by separately considering the cases A>0 and A<0 and determining which branch is selected by \min(\cdot) . ∎ C.3 Gradient of the Importance Ratio Lemma C.2 (Ratio gradient) . Since \pi_{\mathrm{ref}} is fixed, the ratio gradient satisfies \nabla_{\theta}r_{i,t}(\theta)=r_{i,t}(\theta)\,\nabla_{\theta}\log\pi_{\theta}\!\left(y_{i,t}\mid x_{i},y_{i,0,\ r_{i,t}(\theta)>1+\epsilon(o_{i}),\\ 0,&A^{\mathrm{hyb}}_{i,t}<0,\ r_{i,t}(\theta)<1-\epsilon(o_{i}),\\ 1,&\text{otherwise}.\end{cases} (27) Proof. In the saturated regions, s(r) is constant and hence has zero gradient. In the unsaturated region, s(r)=rA , and the claim follows from Lemma C.2 . ∎ C.5 Gradient of the KL Regularizer Let h_{i,t}=(x_{i},y_{i,1 , fork branches using top- B_{t} tokens. 7: Expand each active branch by one token under \pi_{\theta}(\cdot\mid x,y_{i,