Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction Junhong Shen*, Hao Bai*, Lunjun Zhang, Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, Aviral Kumar Abstract Abstract: The current paradigm of test-time scaling relies on generating long reasoning traces (“thinking” more) before producing a response. In agent problems that require interaction, this can be done by generating thinking traces before acting in the world. However, this process does not allow agents to acquire new information from the environment or adapt their behavior over time. In this work, we propose to scale test-time interaction , an untapped dimension of test-time scaling that increases the agent’s interaction horizon to enable running rich behaviors such as exploration, backtracking, and dynamic re-planning within a single rollout. To demonstrate the promise of this scaling dimension, we study the domain of web agents. We first show that even prompting-based interaction scaling without any training can improve task success on web benchmarks non-trivially. Building on this, we introduce TTI (Test-Time Interaction), a curriculum-based online reinforcement learning (RL) approach that trains agents by adaptively adjusting their rollout lengths. Using a Gemma 3 12B model, TTI produces state-of-the-art open-source, open-data web agents on WebVoyager and WebArena benchmarks. We further show that TTI enables agents to balance exploration and exploitation adaptively. Our results establish interaction scaling as a powerful, complementary axis to scaling per-step compute, offering new avenues for training adaptive agents. Project page: https://test-time-interaction.github.io Code: https://github.com/test-time-interaction/TTI 1 Introduction Recent advances in foundation models have enabled a shift from static language models to interactive agents that perform multi-step tasks in dynamic environments like browsers [ 1 , 2 , 3 , 4 , 5 , 6 ] , terminals [ 7 ] , and the physical world [ 8 , 9 , 10 , 11 , 12 , 13 ] . These agents operate in closed-loop settings where each action changes the current state of the world and affects future interaction with the environment. As a result, interactive agents must plan under uncertainty and adapt to failures in real time to be successful. How can we build agents that succeed in such interactive settings? Current post-training approaches produce reactive agents that respond to immediate observations but struggle with evolving or uncertain task dynamics. Methods like supervised fine-tuning (SFT) on expert demonstrations [ 14 , 15 , 16 , 17 , 18 ] or reinforcement learning (RL) with task rewards [ 19 , 20 , 21 , 22 , 23 ] typically train agents to predict a single best action at each step. Even with test-time scaling, where agents are prompted to “think” longer before prescribing an action [ 24 , 25 , 26 ] , they are still optimized to select the most effective action based on the agent’s internal state. While sufficient for fully observable and stationary tasks, reactive policies based on the agent’s internal estimate of the task state are often suboptimal in partially observable (e.g., incomplete details visible on a page) or non-stationary (e.g., fluctuating prices during flight booking) settings, where adaptive, information-seeking behavior is critical. In this paper, we argue that instead of reactive “optimal” policies, agents should learn adaptive policies that can collect new information from the environment and adjust their behaviors on-the-fly. A pre-requisite for such adaptability is the ability to take more actions during deployment than those prescribed by an expert trajectory. We therefore propose a new dimension of test-time scaling: increasing the number of interaction steps of the agent . This allows agents to have sufficient context and time to attempt different behaviors. For example, in a hotel booking task, an agent must first browse many listings to compare user reviews and check availability before selecting the best option. Interaction scaling is orthogonal to existing methods based on chain-of-thought (CoT), which emphasize deeper reasoning per step but do not support information-gathering from the environment. This notion of information gain is unique to agentic tasks with partial observability and requires interaction, not merely larger per-step compute. For instance, an agent that reasons deeply about one selected hotel without interacting further may miss better options that show up only after exploration. Although the idea of interaction scaling is conceptually straightforward, extending it to post-training and teaching agents to scale interaction autonomously presents key challenges. Without appropriate training signals, agents may overfit to exploratory behaviors like blindly clicking links but not making progress toward the actual task objective, wasting the additional steps. To tackle this issue, we propose to combine online RL with a curriculum that prescribes how to scale the interaction horizon, training agents that first learn effective exploitation before extending their horizon to explore. We instantiate our approach in the domain of web agents, a widely applicable setting with well-established benchmarks. We first show that scaling test-time interaction via prompting the agent to “think and act again” after it decides to terminate can already improve the task success rate from 23% to \geq 28% on WebArena [ 2 ] (see Figure 2 for details). While this increases trajectory length and the number of tokens generated, spending an equivalent amount of compute on conventional test-time scaling methods like forcing the agent to think for longer [ 27 ] or running best-of- n [ 28 , 29 , 30 ] yields less than a 3% gain. These findings validate interaction scaling as a promising and complementary axis of test-time scaling. We then move beyond prompting and develop TTI (Test-Time Interaction), a curriculum-based RL approach that trains agents to adaptively scale interaction by gradually increasing the rollout horizon. We scale TTI to large and diverse training sets ( > 100K tasks across \sim 20 domains) by integrating it with an automated pipeline that generates synthetic tasks for online data collection. TTI achieves state-of-the-art performance among open-source agents trained on open data on both WebVoyager [ 1 ] and WebArena [ 2 ] , using only a 12B Gemma 3 model, improving over the non-fine-tuned agent by 9% and 8%, respectively. Our analysis further shows that curriculum training enables adaptive exploration: agents learn to initiate new searches or backtrack in complex tasks, while following efficient paths in simpler ones. In summary , we introduce interaction scaling as a new dimension for test-time scaling of agents. We propose TTI to train agents by adjusting interaction horizon dynamically. Our results show that TTI yields strong empirical gains and offers promising directions for domains beyond web navigation. 2 Related Work Scaffolded foundation models as web agents. Prior works use external control structures to scaffold foundation models via modular prompting [ 31 , 6 , 32 , 33 , 34 , 35 , 36 , 37 ] , programs [ 38 , 39 , 40 , 41 , 42 ] , or feedback mechanisms [ 43 , 44 , 45 , 46 ] . These methods often rely on proprietary models like GPT-4 [ 47 ] or Claude [ 48 ] . Thus, progress is driven by designing better prompts and workflows for planning [ 49 , 50 , 51 , 52 ] , self-correction [ 53 ] , self-evaluation [ 54 , 55 ] , or by integrating external modules such as memory [ 56 ] or retrieval systems [ 57 ] . More recently, developing specialized agents has become a promising direction. ScribeAgent [ 14 ] first uses real-world data to demonstrate that simple fine-tuning can outperform most prompt-based agents. Prior work also builds automated data curation workflows [ 58 , 59 , 60 ] and distillation methods [ 18 ] . Despite these efforts, scaffolding methods remain fundamentally limited: they do not enable agents to self-improve through interaction, and rely on fixed wrappers that lack adaptability across diverse environments. RL training for foundation model agents. RL-based approaches enable agents to autonomously improve through interaction. Prior work has explored DPO [ 61 ] , actor-critic [ 20 , 21 , 62 ] , or distributed sampling [ 63 ] . Pipelines like PAE [ 19 ] and Learn-By-Interact [ 64 ] support automatic task generation, exploration, and labeling. However, most of these approaches lack mechanisms for test-time exploration, limiting the agent’s ability to adapt its behavior over long horizons, especially under partially observable conditions. As Bai et al. 2024 note, continued training after deployment is often required just to maintain performance with these methods. Our work addresses this limitation by scaling test-time interaction as an independent dimension, allowing agents to refine behavior while acting. Curriculum-based RL has been applied in AutoWebGLM [ 65 ] and WebRL [ 23 ] , where curricula are based on estimated task difficulty derived from the complexity of LLM-generated instructions. While our approach also induces a progression from simpler to more complex behaviors, it does so by gradually increasing the allowed interaction horizon rather than relying on explicit measures of task difficulty. Scaling test-time compute. Increasing test-time compute via best-of- n sampling [ 29 ] , beam search [ 66 , 67 ] , or verifiers [ 68 , 69 , 70 ] has shown to improve performance in reasoning-heavy tasks. In non-interactive settings like math and competitive coding, recent methods train models to generate long CoT and scale reasoning internally [ 71 , 27 , 72 , 73 , e.g.,] . As for multi-turn interactive settings, most existing works simply integrate CoT prompting into the agent system to enhance per-step reasoning [ 74 , 52 , e.g.,] . EXACT [ 75 ] scales up the search process for each action, GenRM-CoT [ 76 ] the number of verifiers, and Jin et al. 2025 the number of agents. However, none of these efforts studies the benefits of scaling over the time horizon, where the agent can explore alternatives, backtrack, or gather more information before committing to certain actions. Our work extends this line of research by introducing test-time scaling of interaction. As we will show in our empirical results (Section 4.2 ), the benefits of scaling test-time interaction go beyond test-time scaling (or “reasoning”) before taking an action, within a given time step, because each extra step of interaction with the environment provides new information to the agentic policy, whereas thinking for longer simply reorganizes information that the agent already has. 3 Problem Setup We consider solving a web task as a finite-horizon sequential decision-making process guided by a reward objective 1 1 1 While this work centers on web agents, we believe the insights should generalize to other agent problem domains, and we hope future work will extend these ideas beyond web agents and web navigation. . Formally, the environment implements a transition function that evolves over time and provides an observation o_{t} at step t reflecting the current task state (details regarding the parameterization of the observation space are shown below). The agent policy \pi is parameterized by a multi-modal foundation model that maps observation history o_{1:t-1} and action history a_{1:t-1} to the next action a_{t} . These histories allow the policy to represent rich, context-dependent behaviors enabled via interaction (details about the design of the action space are shown below). We denote the environment horizon, or the maximum number of interaction steps allowed in the environment, as h . For each task, the actual interaction process ends when the agent issues a stop signal or reaches the step limit h . Let h_{\mathrm{stop}}\in(0,h] denote the actual number of steps taken. The agent receives a reward of 1 for task success, and 0 otherwise. Observation space design. Following [ 14 , 19 , 2 ] , we consider an observation space consisting of the task goal, the URL, and a structured representation of the current web page that includes both the accessibility tree of the web page and a screenshot augmented with a set-of-marks overlay [ 1 ] , which assigns a unique identifier to each element that the agent can interact with. While the agent has access to all past observations in principle, doing so quickly exhausts the context window in practice, so we truncate observation history to the most recent three steps, similar to prior works [ 19 ] . However, the agent still has access to all of its past actions. We show in Appendix B an example observation. Action space parameterization. We adopt a discrete action space with six actions: click, type, scroll, go back, search (e.g., Google or Bing), and stop the task with an answer. Following Yang et al. 2024 , we do not consider compound actions like goto[url] , as complex action spaces can hinder performance. For a detailed definition of each action, see the agent prompt in Appendix D.1 . 4 Scaling Test-Time Interaction: A New Dimension of Agent Scaling Prior methods for LLM test-time compute scaling usually scale the number of thinking tokens at each step [ 78 , 79 , 80 , 73 ] , but this does not enable an agent to engage in longer interactions with the environment to collect new information. In principle, scaling the maximum number of interaction steps should allow the agent to employ richer behavioral strategies such as re-attempts, backtracking, and recovery. We will now verify this hypothesis via controlled experiments on WebArena [ 2 ] . We will then build upon these insights to develop TTI , an online RL method to explicitly train agents to optimize test-time interactions. Controlled experiment setup. We choose WebArena [ 2 ] as our testbed, primarily because it enables reproducible interaction with diverse domains (OneStopShop, Reddit, GitLab, CMS, and OpenStreetMap) and is equipped with ground truth evaluators. We randomly sample 62 tasks for testing and reserve the remaining 750 for online training (see Section 5.1 ). To ensure sufficient interaction, we set a generous test-time limit of h=30 , which is well above the average length of around 6 steps required by most tasks [ 56 , 81 ] . Note that experiments in this section are for analysis rather than benchmarking purposes. We will show more experimental results on multiple benchmarks in Section 6 . Table 1 : Base results averaged over three runs. Prompt Task SR (%) Action Only 14.76 CoT 23.81 To study the effect of increasing h , we use a simple prompting-based agent with Gemma 3 12B [ 82 ] base model, which observes the web page and outputs an action via a single model call. It does not leverage any retrieval, verifiers, or other external modules, ensuring any performance gains come solely from increased h but not auxiliary scaffolding. We also study whether it is beneficial to prompt the agent to generate a reasoning trace before acting (see Appendix D.1 for the templates). As Table 1 shows, CoT prompting yields significantly higher task success rate (SR) than direct action generation, setting a baseline of 23.81% on the test tasks. While we also tried more complex prompts that explicitly ask the agent to summarize the state, assess progress, and reflect, they did not yield significant gains (see Appendix D.3 ). We thus adopt a simple chain-of-thought (CoT) approach as the default prompting strategy for experiments in this section. 4.1 Scaling Test-Time Interaction by Acting Longer Figure 1 : Scaling test-time interaction by prompting the agent to “re-check” its answer. More re-checks lead to longer trajectories (dots) and higher task success rates (bars), indicating at least some correlation between acting for longer and higher success rates. To study the impact of test-time interaction scaling, we introduce a purely inference-time “check-again” mechanism: after the agent issues the task completion action, we explicitly prompt it to reconsider its decision by “ You just signaled task completion. Let’s pause and think again… ” We can extend re-checking from double-check (two passes) to triple-check (three passes) and beyond, using slightly varied prompt phrasings for each pass. Detailed prompts are in Appendix D.4 . As shown in Figure 1 , prompting the agent to re-check not only increases the actual interaction length h_{stop} (line plots), but also improves the success rates on most WebArena domains (bar plots). When being asked to “check-again”, the agent either reaffirms its decision (e.g., “ I previously stated the driving time was approximately 30 minutes….30 minutes seems plausible with typical traffic conditions. I’ll stick with my previous answer. ” ) or revises it upon reflection (e.g., “ My apologies. I jumped to a conclusion prematurely. Although the address book *displays* the desired address, the task requires me to *change* it….I should click button [24] to modify the address. ” ). In particular, it changes its action \sim 25% of the time after double-checking. This highlights the potential of interaction scaling: when given sufficient time, the agent is likely to explore alternatives before reaching an answer. The chances of the answer being correct could thus be higher. However, we do observe that repeatedly prompting the agent to re-check can sometimes lead to confusion or hallucination, causing it to revise correct answers into incorrect ones. This may explain the performance drop observed in domains like Map. This is perhaps an inevitable limitation of scaling test-time interaction via prompting alone, akin to how prompting is not an effective way to even scale per-step test-time compute (see self-correction results for prompted models in Qu et al. 2024 ). We discuss this limitation further in Section 4.2 and address by training the agents explicitly. 4.2 Scaling Test-Time Interaction vs. Per-Step Test-Time Compute Next, we examine the effect of scaling interaction compared to scaling per-step reasoning: Given a total token budget, should agents prioritize more interaction steps or generating longer reasoning traces at each step? To explore this, we study two conventional test-time compute scaling methods. Per-step budget forcing. Following Muennighoff et al. 2025 , we prompt the agent to “wait and think again” after generating an initial CoT, encouraging more intermediate reasoning before it commits to an action. We vary the number of forced waits from 1 to 4. Despite trying various prompts to induce longer thinking from the agent (see Appendix D.4 ), our agent changes its actions only 15% of the time, and most of these changes involve reordering subtasks rather than exploring alternative paths or error correction. Per-step best-of- n . At each step, we sample n\in\{3,5,7\} candidate actions and select one via majority voting, similar to [ 14 ] . We did not scale n up further because best-of- n is compute-intensive for multi-step settings ( n\cdot{h} more expensive than the baseline per rollout), and we observe diminishing returns from sampling more actions, despite tuning sampling hyperparameters such as the temperature. Finding 1: Gaining new information via interaction beats thinking more a single step. Figure 2 (top) plots the task success against total compute, measured by the number of tokens per trajectory in log scale. Among the three strategies, interaction scaling ( green ) shows the steepest upward trend, achieving Figure 2 : Top: Task success rate vs. total compute cost (measured in tokens per trajectory, log scale). Interaction scaling consistently achieves the highest success rate for a given compute budget. Bottom: Decomposition of compute into tokens per step and number of interaction steps (excluding the initial point for each approach). Interaction scaling achieves superior performance by distributing compute across more steps with lower per-step cost, in contrast to methods that put more tokens in fewer steps. the highest success rate as the allowed token budgets increase. Budget forcing ( blue ) yields moderate gains but plateaus around 0.26. Despite incurring the highest cost, best-of- n ( orange ) brings the least improvements, suggesting that repeatedly sampling actions per step is a less effective use of compute in interactive tasks. A natural question that arises here is: how should we distribute a bounded compute budget between running more interaction steps vs. reasoning longer? These two dimensions present different costs per unit, which may not be known apriori. Hence, we illustrate how scaling the number of interaction steps and the number of tokens per step affect performance and the total number of tokens individually. Figure 2 (bottom) decomposes total compute into tokens per step (y-axis) and steps per rollout (x-axis), so the shaded areas indicate the average total compute required per trajectory. Interaction scaling extends along the x-axis, while per-step reasoning scales along y-axis. We find that scaling across steps is more effective than scaling within steps in WebArena tasks (Figure 2 , top), likely because the former enables the agent to gather new information and enrich its context. This ability to query and observe external feedback is unique to agentic settings but not single-turn question-answering tasks. While standard per-step reasoning is constrained by the information already available at each step, our approach takes advantage of this dynamic interaction. However, in principle, one can combine more reasoning per-step with more interaction, at the cost of spending many tokens. Finding 2: Prompting alone is insufficient for interaction scaling. While our results highlight the potential of scaling interaction, the “check-again” strategy only allows the agent to revisit its behavior upon task completion, it does not enable it to implement nuanced behaviors such as switching between exploration and exploitation in the middle of a rollout. We also experimented with combining interaction scaling with budget forcing and best-of- n (Appendix Table 6 ) and observe that simply increasing test-time compute via prompting does not yield additive gains. In fact, the final downward trend in Figure 2 (top) suggests that asking the agent to re-check too many times or think for too long can confuse it and degrade performance. This shows the need for methods that train agents to optimize for best behavior when scaling test-time interaction, rather than naïve prompting. Takeaways: Scaling test-time interaction vs. test-time compute Under a fixed compute budget (measured by total tokens), gaining information through longer interaction with the environment can be more effective than solely deepening per-step reasoning. While longer chain-of-thought can improve local decision quality, interaction scaling offers a complementary and often more compute-efficient way to enable agents to adapt and explore over longer horizons. 5 TTI : Curriculum-Based Online RL for Scaling Interaction How can we extend beyond prompting-based scaling to training agents that can effectively utilize interaction scaling? A natural starting point is to draw inspiration from current approaches for optimizing test-time compute [ 30 , 72 , 71 ] and extend these ideas to interactive settings. Specifically, we can run reinforcement learning (RL) with binary task rewards and longer task horizons. However, is this approach sufficient? We first describe the key challenges in learning to scale test-time interaction, and then develop our approach to address them via curriculum learning. 5.1 Challenges of Training Agents with Fixed, Long Horizons A natural way to encourage the agent to learn to take more steps is to train at long horizons. To study this, we can run the simplest form of REINFORCE [ 84 ] with binary rewards R(\cdot) , also known as online filtered behavior cloning (BC) or online STaR [ 20 , 85 ] . Only successful trajectories are retained, and the agent is updated by maximizing the log-likelihood of actions conditioned on those high-reward rollouts: \displaystyle\argmax_{\theta}\penalty\ \penalty\ \mathbb{E}_{\mathcal{T}\sim\text{tasks}}\left[\mathbb{E}_{o_{0:h},a_{0:h-1}\sim\pi(\cdot\mid\mathcal{T})}\left[\left(\underbracket{\sum_{t=0}^{h-1}\log\pi_{\theta}(a_{t}\mid o_{\leq t},\mathcal{T})}_{\text{likelihood of trajectory}}\right)\cdot\mathbbm{1}\left[\underbracket{R(o_{0:h},\mathcal{T})}_{\text{reward}}=1\right]\right]\right] (5.1) We use filtered BC as it is stable throughout training (no negative gradient [ 86 ] ), has been utilized previously for training agents [ 20 ] , and is a good “first-choice” algorithm for studying the role of interaction scaling. We scale the agent’s horizon on WebArena, varying h\in\{5,10,20\} . Smaller h exposes the agent only to exploitative rollouts that succeed within allowed time steps, while larger h also includes more exploratory rollouts. We use the non-test tasks for rollout. Details are in Appendix D.5 . As shown in Figure 3 (left), the agent trained with h=5 learns quickly, likely because on-policy RL is more sample-efficient at smaller horizons, but it also quickly overfits, and performance decreases with more training (x-axis) 2 2 2 We use the same training hyperparameters (Appendix D.5 ) across all settings for fair comparison. While reducing the learning rate can help reduce performance drop, it does not address the core issue with small horizons: fewer successful rollouts and shorter trajectories lead to significantly less training data. . This agent often terminates prematurely during evaluation despite being allowed to interact for much longer time. Conversely, agents trained at longer horizons generally learn policies that are quite stochastic and learn significantly more slowly due to higher variance of the loss and credit assignment challenges due to longer horizons [ 87 , 88 , 89 , e.g.,] . We manually inspect the trajectories and find that the h=20 agent tends to associate exploratory actions such as “going back” or “trying random links” with high rewards initially. Noisy credit assignment with h=20 slows learning, and only after several iterations do the agents begin to recover and produce more robust policies. The impact of horizon is domain-dependent: in complex domains requiring exploration (e.g., CMS), long-horizon agents outperform, while in simpler settings (e.g., Reddit), performance differences are minimal. We also note that the total number of tokens generated per action slightly decreases throughout training, indicating that the training does not incentivize increasing the length of reasoning directly. Importantly, although the interaction length increases as expected for h=20 (Figure 3 , right), noisy credit assignment and slower learning suggests that simply setting h to be large is insufficient to learn to scale test-time interaction reliably. These observations motivate our method’s core idea: rather than fixing the horizon throughout training, our approach aims to scale their interaction length dynamically. Figure 3 : Online RL with different values of maximum interaction horizon. Left: success rates for different domains. “Harder” means generally lower success rate. Right: average rollout length ( h_{\mathrm{stop}} ) on the evaluation set. 5.2 Our Approach: Curriculum Over Interaction Horizon To address these challenges, we propose TTI (Test-Time Interaction), a curriculum-based online RL approach that trains the agent with short trajectories initially and gradually exposes it to longer ones. Existing curriculum learning methods in RL [ 90 , 91 , 92 , 93 , 94 , e.g.,] or web agents [ 23 , 65 ] typically prioritize easier tasks before harder ones, often relying on predefined heuristics or external measures of task complexity. In contrast, we define curriculum progression in terms of the maximum number of steps an agent is allowed per trajectory. While interaction length can serve as a rough proxy for task difficulty, our approach does not require explicit labeling or estimation of task complexity. How do we design a curriculum over the interaction horizon h ? Ideally, the curriculum should allow the agent to first learn basic, “atomic” skills to solve easier tasks, then progressively tackle complex ones via skill chaining and exploration. To enable this kind of behavior, we explored two strategies: (1) a conservative, additive increase in h per iteration, giving the agent sufficient time to solidify core task-solving skills; and (2) a more aggressive, multiplicative increase, which assumes the agent can quickly acquire the basic skills and benefit from earlier exposure to exploratory behavior. Formally, for iteration i : \displaystyle\footnotesize h_{i} \displaystyle:=\text{clip}(h_{\min}+i,\;h_{\max})\quad\text{(Additive curriculum)} (5.2) \displaystyle h_{i} \displaystyle:=\text{clip}(h_{\min}\cdot i,\;h_{\max})\quad\text{(Multiplicative curriculum)} (5.3) We store the rollouts in a replay buffer and assign higher weights to more recent trajectories. The full pseudocode for TTI and implementation details are provided in Appendix C . Table 2 : Comparing various curricula. A multiplicative curriculum produces the best success rate. Schedule Task SR (%) Additive 29.50 Multiplicative 32.25 Empirical insights. We instantiate these two strategies in WebArena, using the non-test tasks for online training. We set h_{\min} to 10 and h_{\max} to 30, and apply the schedules on top of filtered BC. Evaluation results after 10 iterations are shown in Table 2 . Multiplicative curriculum outperforms the additive one, possibly because it exposes the agent to longer horizons early on and helps prevent it from overfitting prematurely to shortcut behaviors like always taking the shortest path. Based on these findings, we adopt the multiplicative curriculum as the default for TTI . Table 2 further shows that even with limited data ( \sim 700 training tasks), TTI outperforms fixed h=20 in Figure 3 by nearly 3%, using 40% fewer training steps over 10 iterations. Next, we demonstrate that this advantage carries over to large-scale online RL training. Takeaways: Curriculum training enables effective interaction scaling Our approach, TTI , gradually scales the horizon using a curriculum, leading to better performance than fixed-horizon baselines. The multiplicative schedule improves both learning efficiency and task success rate. 6 Experiments: Scaling Up to Realistic Benchmarks We now provide a comprehensive evaluation of TTI in large-scale, realistic environments. More generally, we demonstrate the effectiveness of training models to make good use of test-time interaction. Figure 4 : Summary of our main results on WebVoyager and WebArena benchmarks . TTI consistently outperforms fixed-horizon training with both short and long horizons. Synthetic task generation and evaluation. To enable large-scale training without training on the benchmark itself, we adopt synthetic task generation inspired by PAE [ 19 ] . Apart from prompting LLMs with seed examples or demos, we also deploy an exploratory agent to freely interact with websites and propose more grounded, diverse tasks. For evaluation, we leverage a prompting-based verifier that uses action histories and screenshots to label rollouts, achieving 88.9% accuracy compared to WebArena’s ground-truth evaluator. Details and prompt templates are in Appendix E.1 . We evaluate agents on (1) WebVoyager [ 1 ] with 427 tasks across 13 domains (we replace Google search with Bing due to ReCaptcha issues); and (2) full WebArena [ 2 ] with 812 tasks. We choose these benchmarks as they are widely used [ 4 , e.g.,] . Training. We obtain 128K synthetic tasks across diverse real-world domains for WebVoyager and 11K tasks for WebArena’s self-hosted domains (data released with the code). We train separate agents for each benchmark to avoid contaminating real-domain agents with synthetic environment data, as WebArena domains still have many differences from their real-world counterparts. We use Gemma 3 12B [ 82 ] as the base model, sampling 512 tasks per iteration for rollouts and updating with 512 successful on-policy trajectories. We apply a multiplicative curriculum with h_{\min}=10 and h_{\max}=30 . We use vLLM [ 95 ] to sample rollouts and use DeepSpeed Zero 3 [ 96 ] with NVIDIA H100 GPUs for training. The evaluator is a Gemma 3 27B model, prompted to detect successful trajectories, which can then be used for the online filtered BC procedure. Other hyperparameters such as the number of iterations, learning rate, and the exact schedule of TTI can be found in Appendix E.3 . Comparisons. We evaluate zero-shot Gemma 3 12B and approaches that utilize a fixed horizon with h\in\{10,30\} . We also compare to closed-source agents (e.g., those based on GPT-4 [ 47 ] and Claude [ 48 ] ), open-weight models trained on proprietary data (e.g., UI-TARS [ 97 ] ), and fully open-weight, open-data models (e.g., PAE [ 19 ] ). A detailed list of the prior approaches can be found in the result tables. 6.1 WebVoyager Results and Analysis State-of-the-art open-weight, open-data performance. We report the overall task success rates (SR) on WebVoyager in Table 3 and Figure 4 (left). The TTI -trained Gemma 3 12B achieves an average SR of 64.8%, setting a new state-of-the-art among open agents trained purely on public data. While previous methods such as UI-TARS achieves a strong SR of 84.8%, they rely on private human-annotated data that remains inaccessible to the open-source community. In contrast, TTI is trained entirely on synthetic data and interestingly, this data is generated by the base model (Gemma 3 12B) itself, meaning that our training protocol implements a form of self-improvement . This shows the practicality of our approach, but also highlights the need for future work on developing high-quality open data comparable to human-generated ones. TTI also obtains the highest SR in 8 out of 13 domains, illustrating its efficacy. Table 3 : WebVoyager results. Training a Gemma3 12B model using TTI attains the best performance among open-weight agents trained on open-source synthetic data in aggregate on the WebVoyager benchmark. Baseline results are taken from Zhou et al. 2024b , Qin et al. 2025 , and illustrate that TTI improves over the best prior open-model, open-source data approach by 30%. Model Average Allrecipes Amazon Apple ArXiv GitHub ESPN Coursera Cambridge BBC Map Search HuggingFace WolframAlpha Proprietary Model Claude 3.7 84.1 - - - - - - - - - - - - - Claude 3.5 50.5 50.0 68.3 60.4 46.5 58.5 27.3 78.6 86.0 36.6 58.5 30.2 44.2 66.7 OpenAI CUA 87.0 - - - - - - - - - - - - - Agent E 73.1 71.1 70.7 74.4 62.8 82.9 77.3 85.7 81.4 73.8 87.8 90.7 81.0 95.7 Open Model, Proprietary Human-Annotated Data UI-TARS-1.5 84.8 - - - - - - - - - - - - - Open Model, Open Synthetic Data LLaVa-34B SFT 22.2 6.8 26.8 23.3 16.3 4.9 8.6 26.8 67.4 16.7 12.2 23.3 20.9 38.1 PAE-LLaVa-7B 22.3 14.3 37.5 17.5 19.0 14.6 0.0 33.3 52.4 18.6 22.5 23.3 19.0 24.4 PAE-LLaVa-34B 33.0 22.7 53.7 38.5 25.6 14.6 13.6 42.9 74.4 39.0 22.0 18.6 25.6 42.9 Gemma 3 12B 55.8 25.7 32.3 45.5 60.6 54.8 60.6 56.3 69.6 65.6 54.8 72.7 66.7 61.1 Fixed h=10 59.1 25.7 74.1 51.5 75.7 70.9 44.1 59.3 66.7 50.0 41.9 60.6 75.5 72.2 Fixed h=30 45.2 20.0 41.9 60.6 42.4 41.9 50.0 34.4 60.6 25.0 29.0 63.6 45.5 69.4 TTI (Ours) 64.8 57.1 48.3 69.6 66.6 45.2 56.3 46.9 85.2 81.2 66.7 72.7 75.7 79.4 TTI outperforms fixed-horizon via adaptive exploration. Table 3 also shows that our curriculum approach outperforms fixed h=10 baseline by 5.7% and fixed h=30 baseline by 19.6% in average accuracy. To better understand the use of interaction within a training rollout, we plot the average number of interaction steps on a held-out validation set with 78 tasks in Figure 5 (a). Note that the agent trained with h=10 learns to continuously reduce the maximum number of steps it spends in a rollout, while h=30 quickly drifts into aimless exploration and executes a larger number of steps pre-maturely in training, hindering performance. This aligns with our findings in Section 5.1 that simply running training at a longer horizon may not be sufficient for obtaining effective interaction scaling. In fact, we find that when training with TTI , the interaction length of the agent’s rollouts first decreases but then starts to increase as the maximum allowed horizon increases, indicating that an adaptive curriculum enables effective interaction scaling. Figure 5 (d) shows that the task success rate also grows over time and correlates with the expanding horizon. While the average task success rates for TTI are better, we observe notable per-domain differences. Figure 5 (e) shows representative per-domain success rates. On domains like Allrecipes and Cambridge, TTI significantly outperforms fixed-horizon and zero-shot approaches, improving success rates by 31.4% and 15.6%, respectively, likely because these domains are highly information-dense and benefit from extended exploration enabled by adaptive interaction scaling. However, in domains like Amazon and GitHub, TTI underperforms the baselines. We notice that the base model already has strong knowledge about domain-specific terminologies (e.g., commit history, forks, stars) in these domains, likely because they are more prevalent than the others, resulting in high base performance. Inspecting the rollouts, we find that instead of using built-in filters and sorting, TTI can engage in exploration behaviors such as initiating Bing searches or consulting external sites. This exposes the agent to noisy or distracting information, reducing task success. We discuss these cases in Section 6.2 . Learning dynamics of TTI . To study how TTI enhances the “within-rollout” exploration capabilities of the agent, we measure the number of GoBack and Bing actions over the course of training. GoBack actions measure the number of retries the agent makes within an episode to get unstuck during exploration. Bing actions correspond to the number of times the agent attempts to seek information by moving to bing.com . As shown in Figure 5 (a, b, and d), the performance of TTI improves substantially as the number of GoBack and Bing actions and the trajectory length grow. Figure 5 : Dynamics of TTI during training. For TTI , the green area represents the phase where the maximum allowed interaction horizon is the largest ( h=30 ), per our multiplicative schedule. All results are evaluated on a held-out subset of WebVoyager, not on the training tasks. a: Average trajectory length, i.e. the average number of steps taken in a trajectory normalized by the average length at the first iteration (iteration 0). b: Ratio of the sum of GoBack and Bing actions out of all actions normalized by the first iteration. c: The average number of tokens each action uses, i.e., the average CoT length during agent reasoning. d: Average task success rates for TTI on the held-out validation set of tasks. e: Per-domain success rates for TTI . All results are evaluated on a held-out subset of WebVoyager tasks. Observe that TTI learns to utilize more interaction steps and explores by calling GoBack and Bing actions once the maximum allowed horizon increases to peak value (i.e., in the green shaded area). The number of tokens appearing every step reduces for TTI compared to the initialization, resulting in significantly shorter CoT compared to the run with h=10 . Also note that the trajectory length and the numbers of GoBack and Bing actions begin to increase with TTI , once the maximum allowed horizon length is increased as a part of the curriculum schedule (this regime is shown by the green shaded area in Figure 5 ). In contrast, these quantities continuously decrease over the course of training for the run with a lower number of maximum interaction steps ( h=10 ). We also find that the trajectory length shoots up substantially for the run with h=30 and this correlates with worse performance. Finally, as shown in Figure 5 (c) we also note that as the agent’s trajectory grows longer with TTI , the number of tokens appearing in per-step reasoning actually becomes smaller. This implies that our agent is automatically learning to tradeoff interaction for per-step compute in order to attain higher performance, and perhaps prevents any issues with overthinking. 6.2 Case Studies: Strengths and Failure Modes We conduct detailed case studies to analyze how TTI behaves across tasks and domains. These cases highlight both the strengths and remaining limitations of our approach. Strength: Effective exploration in complex tasks (example visualized in Appendix E.4 ). For complex, exploratory tasks that require information retrieval, TTI trains the agent to extend its interaction horizon through search and backtracking, thus gathering and comparing information before making decisions. For instance, when tasked to find the baking temperature of an apple pie recipe with 4+ stars and 50+ reviews, our agent first selects a recipe but encounters a pop-up it cannot dismiss due to backend issues. It then tries another recipe but finds no baking temperature. Returning to the listing again, it correctly identifies one that meets all criteria. We also observe that such behaviors emerge progressively. In early training with shorter horizon, TTI -agent tends to stick to the first recipe it finds, keeps retrying it and saying “I remember seeing one with 613 ratings earlier” instead of seeking alternatives. Only after the training horizon becomes longer does it learn to explore and backtrack. This illustrates that when TTI runs a curriculum over interaction length, it teaches agents to adjust their horizon within a task and shift from exploitation to exploration. In contrast, training with a fixed short horizon can make it difficult to develop such exploratory behaviors. Strength: Strategic exploitation in simple tasks (Appendix E.5 ). For simpler tasks with clear, deterministic paths (e.g., form filling or direct lookups), TTI -agent completes tasks efficiently without over-exploration. For example, when instructed to find the “top trending open-source project on machine learning” in GitHub, the agent goes directly to the Open-Source menu, selects the Trending tab, and performs search. This shows that TTI balances exploration and exploitation based on task context. Despite these strengths, we also observe characteristic failure modes that point to areas for improvement and may partly explain the agent’s lower performance on domains like GitHub. Failure mode: over-reliance on resets (Appendix E.6 ). When an action fails, our agent can reset the task by returning to the Bing search page rather than attempting recovery within the target domain. This suggests the agent treats search as a universal fallback, even when more domain-specific actions (e.g., revisiting menus, refining filters) would be more effective. We also observe repeated resets within the same trajectory, indicating a lack of adaptive error recovery. While agents can extend horizons through both resetting and backtracking, the latter is often more natural. This highlights an area where TTI could improve by guiding exploration more systematically and enforcing structure. We believe that using dense rewards [ 72 , 69 ] or running full multi-turn RL with value functions [ 98 ] may address this issue. Failure mode: limited self-verification (Appendix E.7 ). We also observe that the agent can fail to verify its actions against the task goal, especially in the last step. In one case, the agent identifies a 2021 GitHub repository for a task requiring one from 2022. While it explicitly acknowledges the mismatch, “It was created in 2021, not 2022, so it doesn’t meet the criteria” , it still submits it as the answer. This implies limited self-verification ability and could be mitigated by longer, more deliberate per-step reasoning. An important next step is to enrich TTI with the cognitive behaviors that enable per-step reasoning [ 99 ] . Table 4 : Full WebArena results. For proprietary agents, we include the top 8 from the official leaderboard. Observe that TTI attains the best performance on an average, with especially better performance on the CMS domain. Method Backbone Average Shopping CMS Reddit GitLab Maps Proprietary-Based IBM CUGA [ 100 ] - 61.7 - - - - - OpenAI CUA [ 4 ] - 58.1 - - - - - Jace AI [ 101 ] - 57.1 - - - - - ScribeAgent [ 14 ] GPT-4o + Qwen2.5 32B 53.0 45.8 37.9 73.7 59.7 56.3 AgentSymbiotic [ 18 ] Claude 3.5 + Llama 3.1 8B 48.5 48.7 41.2 63.2 47.2 57.8 Learn-by-Interact [ 64 ] Claude 3.5 Sonnet 48 - - - - - AgentOccam-Judge [ 33 ] GPT-4 45.7 43.3 46.2 67.0 38.9 52.3 WebPilot [ 34 ] GPT-4o 37.2 36.9 24.7 65.1 39.4 33.9 Fully Open-Source Learn-by-Interact [ 64 ] Codestral 22B 24.2 - - - - - (Self-Improvement) AgentTrek [ 37 ] Qwen2.5 32B 22.4 - - - - - AutoWebGLM [ 65 ] ChatGLM3 6B 18.2 - - - - - NNetnav [ 102 ] Llama 3.1 8B 7.2 7.4 4.2 0 0 28.5 Zero-Shot Baseline Gemma 3 12B 18.3 26.7 8.7 30.9 5.5 27.7 Fixed h=10 Gemma 3 12B 23.8 28.4 15.6 26.0 13.2 34.7 Fixed h=30 Gemma 3 12B 19.0 25.7 9.7 29.8 8.7 28.57 TTI (Ours) Gemma 3 12B 26.1 33.9 15.5 35.3 15.7 40.5 6.3 WebArena Results and Analysis We further assess TTI on the full WebArena [ 2 ] (we only use the prompting-based verifier for training, but use the original benchmark evaluators for evaluation). As shown in Table 4 and Figure 4 (right), TTI obtains the highest performance among open-source agents trained entirely using a self-improvement procedure on self-generated data, without relying on proprietary models for task completion or distillation. While TTI improves over the zero-shot baseline by 7.8%, the gains are smaller than on WebVoyager, possibly because: (1) WebArena tasks are more complex, as reflected in lower accuracies even for proprietary models, leading to fewer successful rollouts per iteration and slower learning; (2) Agents occasionally attempt actions that are valid in real-world domains but invalid in WebArena counterparts (for example, search works reliably on Reddit but fails in WebArena’s Postmill due to environment bugs). More experiment details are in Appendix D.6 . Figure 6 : We apply test-time re-checks to TTI checkpoints. Note that inference-time re-checking improves performance on all TTI checkpoints, and more importantly, training further with TTI improves performance more than inference-time re-checks as expected. Further scaling. While TTI equips agents with the ability to adjust their interaction horizon during deployment, an open question remains: Can we further amplify performance by combining TTI with inference-time interaction scaling techniques such as re-checking as discussed in Section 4 ? To explore this, we apply the “check-again” strategy (Section 4 ) to intermediate TTI checkpoints. Due to the high evaluation cost associated with evaluating on full WebVoyager or WebArena, we leverage the WebArena subset checkpoints obtained in Section 5.2 . As shown in Figure 6 , applying re-checking on top of TTI improves task success across various training stages. The benefits are more obvious in the early stages of training, when the agent has a stronger bias to terminate prematurely. As training progresses, TTI encourages longer interaction traces that naturally incorporate behaviors like re-checking, reducing the added benefit of explicit re-checks. Nonetheless, even in later stages, re-checking does continue to provide modest gains in performance, serving as a safety-check for well-trained agents. 7 Conclusion and Future Work In this work, we introduced interaction scaling as a new dimension of test-time scaling for interactive agents. Through empirical studies on web agents, we validate that interaction scaling enables agents to explore and adapt dynamically, significantly improving task performance. Despite the promising results, there are a number of avenues for future work which we list below. • Extension to other agentic problem domains. While we only study the effects of interaction scaling on web environments, we believe that this procedure is likely going to be even more effective in domains that admit more stochasticity and uncertainty, such as robotic control or open-world compute use problems. In these settings, solving tasks would require gather information first before attempting to solve the task. By utilizing interaction as an effective tool for information gathering, we can attain much better performance. • Balancing the tradeoff between thinking more and acting more. While we illustrate that spending a given amount of total token budget on acting for longer can be more effective than reasoning for longer, an interesting open question is how we should incentivize RL training to best balance thinking and acting during the process of learning. In our experiments, we observed an increased preference towards acting longer, with the number of tokens in per-step reasoning decreasing compared to the initialization. While this results in better use of total overall test-time compute given our results in Section 4 , it is unclear as to what a general thumb rule and training procedure should be to attain the best tradeoff between acting longer and thinking more. • Improved RL algorithms for powering interaction scaling. The RL approach we utilize in this work is extremely simple as it corresponds to online filtered behavior cloning (BC). An immediate next promising direction is to extend TTI to utilize negative gradients [ 86 ] via GRPO [ 78 ] or PPO [ 103 ] . However, stochasticity and, even non-stationarity, in interactive agent environments suggests that RL algorithms that train value functions [ 21 , 98 ] are likely to be more successful at effective credit assignment as the task horizon is scaled further. In addition, we will need to address challenges pertaining to memory and context length, that is very likely to overflow as we scale horizon even further. Tackling any of these challenges would be exciting for future work. Appendices Appendix A Broader Impact This work contributes to the development of more adaptive and capable AI agents by introducing a new test-time scaling dimension focused on interaction rather than per-step reasoning alone. While this approach improves robustness and generalization in open-ended environments, it also raises important considerations. Increased agent autonomy can amplify both the benefits and risks of deployment in real-world systems. Moreover, agents capable of richer behaviors could be applied to sensitive domains (e.g., customer service, education, or automation workflows) where unintended actions could have large impacts. We encourage future work to consider ethical safeguards, interpretability tools, and human-in-the-loop designs when deploying interaction-scaled agents. Our experiments are conducted entirely in simulated environments, and we hope this work inspires further research on controllable and trustworthy agent behavior under realistic constraints. Appendix B Observation Space Design We use the screenshot accompanied with the web page’s accessibility tree as our main observation. We study two versions of accessibility tree. Rich accessibility tree is modified from the WebArena code and looks like: [21]: RootWebArea ’Dashboard / Magento Admin’ focused: True; [0]: link ’Magento Admin Panel’; [1]: link ’DASHBOARD’; [2]: link ’SALES’; [3]: link ’CATALOG’; [4]: link ’CUSTOMERS’; [5]: link ’MARKETING’; [6]: link ’CONTENT’; [7]: link ’REPORTS’; [8]: link ’STORES’; [22]: link ’SYSTEM’; [23]: link ’FIND PARTNERS & EXTENSIONS’; [24]: heading ’Dashboard’; [9]: link ’admin’; [10]: link ”; [25]: StaticText ’Scope:’; [12]: button ’All Store Views’ hasPopup: menu; [13]: link ’What is this?’; [14]: button ’Reload Data’… Simple accessibility tree is modified from the PAE code and looks like: [1]: "Dashboard"; [2]: "Sales"; [3]: "Catalog"; [4]: "Customers"; [5]: "Marketing"; [6]: "Content"; [7]: "Reports"; [8]: "Stores"; [9]: "admin"; [12]: