DORA Explorer: Improving the Exploration Ability of LLMs Without Training
Priya Gurjar, Md Farhan Ishmam, Kenneth Marino · 2026
Read DORA Explorer: Improving the Exploration Ability of LLMs Without Training by Priya Gurjar, Md Farhan Ishmam, Kenneth Marino (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17244.txt.
Introduction
Gurjar, Ishmam, and Marino present DORA Explorer, a training-free method boosting agent search. Scaling temperature spreads diversity at the token level, not the sequence level, leaving thin search, weak answers, and repeat actions. DORA makes trial actions, rates them with sequence-level scores, and samples one via a tunable knob. In Multi-Armed Bandit tests, DORA clearly beats temperature-based sampling. On TALES, DORA gives gains, lifting Qwen-2.5 7B in TextWorld from about 31 percent with ReAct to about 46 percent. DORA also prevents loops.
Catalog record
| CREATOR | Priya Gurjar, Md Farhan Ishmam, Kenneth Marino |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50114 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17244
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DORA Explorer: Improving the Exploration Ability of LLMs Without Training Priya Gurjar, Md Farhan Ishmam, Kenneth Marino Abstract Despite the rapid progress, LLMs for sequential decision-making ( i.e. LLM agents) still struggle to produce diverse outputs. This leads to insufficient exploration, convergence to sub-optimal solutions, and becoming stuck in loops. Such limitations can be problematic in environments that require active exploration to gather information and make decisions. Sampling methods such as temperature scaling introduce token-level randomness but fail to produce enough diversity at the sequence level. We analyze LLM exploration in the classic Multi-Armed Bandit (MAB) setting and the Text Adventure Learning Environment Suite (TALES). We find that current decoding strategies and prompting methods like Chain-of-Thought and Tree-of-Thought are insufficient for robust exploration. To address this, we introduce DORA Explorer (Diversity-Oriented Ranking of Actions), a training-free framework for improving exploration in LLM agents. DORA generates diverse action candidates, scores them using token log-probabilities, and selects actions using a tunable exploration parameter. DORA achieves UCB-competitive performance on MAB and consistent gains across TALES, e.g. , improving Qwen2.5-7B’s performance from 29.2% to 45.5% in TextWorld. Our project is available at: https://dora-explore.github.io/ . 1 Introduction The strong reasoning capabilities of auto-regressive LLMs have led to their widespread adoption as decision-making agents across diverse domains, including recommendation systems ( 34 ) , scientific discovery ( 21 ) , game-playing ( 3 ) , healthcare ( 7 ) , and robotics ( 14 ) . Despite the great zero-shot capabilities of LLM agents on these tasks ( 12 ) , the auto-regressive nature of most language models introduces a new problem: exploration . Exploration is a fundamental issue in decision-making problems and classical reinforcement learning (RL). In a new environment, the agent often operates under partial information and must balance exploration to acquire knowledge with exploitation to maximize reward or goal completion. Exploration enables agents to gather information about unknown states, avoid suboptimal local policies, and discover high-reward behaviors that are not immediately apparent. Despite the importance, exploration is an area where LLM agents struggle ( 15 ; 26 ) . In environments with text interfaces, LLM agents use their language output space to produce domain-specific actions in that environment. For example, in a text adventure game, an agent might output “move forward” which can then be interpreted by the environment to move the player character forward. While this translation allows for LLMs to work zero-shot ( 12 ) , the text-based nature of this action space means that to properly explore, LLM agents must sample diverse sequences of tokens which comprise the action. Unfortunately, as seen in narrative generation ( 20 ) , sampling diverse sequences is extremely challenging and LLMs continue to generate high-likelihood and uncreative sequences. In the context of LLM agents, this often results in repetitive or high-confidence action patterns ( 15 ) . Temperature sampling is a common strategy to increase diversity by introducing stochasticity in the distribution of the output tokens. However, a sequence of such tokens need not represent a novel text; indeed, recent empirical studies by 20 suggest that temperature (often cited as a ’creativity parameter’) has only a weak correlation with novelty in narrative generation. Furthermore, 23 found that increasing sampling temperature also reduces task accuracy compared to deterministic decoding. In this work, we introduce DORA Explorer (Diversity-Oriented Ranking of Actions), a training-free inference time framework designed to improve exploration in LLM-based agents, visualized in Figure 1 . While prior approaches have attempted to improve LLM agent exploration using pure sampling ( 6 ) , pure prompting ( 31 ) or through controlled exploration ( 28 ) , our approach uses aspects of all three approaches to explicitly control the exploration/exploitation tradeoff, generate a variety of diverse actions, and sample actions based on the level of exploration through an exploration parameter \lambda . LLMs tend to favor a narrow set of familiar responses over other plausible alternatives ( 33 ) ; however, when explicitly prompted to generate multiple candidates, they can help recover some of this latent diversity ( 33 ) . DORA leverages this property for different candidate action generation. For sampling these action candidates in proportion to the desired level of exploration \lambda , we use a combination of sequence-level metrics, mean log prob ( 16 ) and variance of log probability ( 8 ) , which reflect the exploratory intent of the action better. Importantly, DORA is modular and plug-and-play: it can be seamlessly integrated into existing LLM pipelines without additional training, environment-specific tuning, or architectural changes. We demonstrate its effectiveness across both Multi-Armed Bandit (MAB) settings and the Text Adventure Learning Environment Suite (TALES), where it consistently improves performance by enabling exploration. DORA achieves UCB-competitive performance on MAB and consistent gains across the TALES environments. Figure 1: (Left) Prompt and temperature-based sampling can fail by repeating previously explored actions and getting stuck in loops. (Right) DORA avoids these pitfalls through a dynamic exploration strategy that decides whether to act greedily or explore based on the exploration degree \lambda , generates candidate actions, scores each one, and samples based on the derived \lambda probabilities. This results in more exploration and better terminal rewards. 2 Related Work We can broadly categorize four approaches to controlling and improving exploration in LLM agent problems: sampling-based methods, prompting-based methods, controlled sampling, and learning-based approaches. DORA Explorer draws on aspects of these first three categories, creating an LLM agent framework that can explicitly control the level of exploration and reliably sample diverse actions without additional training. The first category of methods is sampling-based decoding strategies for LLMs. Top- k sampling restricts generation to the k most probable tokens ( 6 ) , top- p (nucleus) sampling ( 10 ) selects the smallest set of tokens whose cumulative probability exceeds a threshold p ( 10 ) , and min- p sampling truncates low-probability continuations relative to the model’s highest-probability token ( 18 ) . These methods are designed to improve fluency, coherence, diversity, and repetition control in open-ended generation. 22 shows that, in LLM-based problem solving on multiple-choice question answering (MCQA) tasks, different sampling strategies exhibit minimal performance difference in the temperature range [0,1] . For higher temperatures ( >1 ), generation quality degrades significantly, leading to incoherent outputs and a steady drop in accuracy. 20 observe only a weak positive correlation between novelty and temperature, and a moderately negative correlation between coherence and temperature. Our approach separates the processes of generating candidates and sampling them to ensure that we have highly diverse but also correct actions, which can also be explicitly controlled through our exploration parameter \lambda . The second type of approach is prompting-based , which includes both frameworks for explicitly increasing the diversity or randomness of LLM-generated outputs and general frameworks for agents in embodied settings to explore more effectively. 15 evaluate 32 prompt designs to study LLM exploration in a Multi-Armed Bandit setting. They find that no matter the prompts used, LLMs do not explore robustly, even for this fairly simple canonical exploration setting. Most relevant to us is 33 , which introduces Verbalized Sampling (VS), a simple prompting technique for generating many diverse output sequences to allow for more diverse outputs (e.g., “Generate 5