Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents

Prahaladh Chandrahasan, Jiahe Jin, Zhihan Zhang, Tevin Wang, Andy Tang, Lucy Mo, Morteza Ziyadi, Leonardo F.R. Ribeiro, Zimeng Qiu, Markus Dreyer, Akari Asai, Chenyan Xiong · 2025

Read Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents by Prahaladh Chandrahasan, Jiahe Jin, Zhihan Zhang, Tevin Wang, Andy Tang, Lucy Mo, Morteza Ziyadi, Leonardo F.R. Ribeiro, Zimeng Qiu, Markus Dreyer, Akari Asai, Chenyan Xiong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05495.txt.

Introduction

Prahaladh Chandrahasan and a large team introduce a platform for judging deep research AI agents, in a 2025 paper. These agents search the web and write long reports, which are hard to grade fairly. Readers need to see not just final text but the steps behind it.

Their platform shows two agents' reports side by side with their working steps exposed. Human raters score the whole report and can also flag single steps or passages. The team adds a simple baseline agent that any large model can plug into. Preference data from seventeen annotators over three agents demonstrates the system. A demo video accompanies the release.

Catalog record

CREATORPrahaladh Chandrahasan, Jiahe Jin, Zhihan Zhang, Tevin Wang, Andy Tang, Lucy Mo, Morteza Ziyadi, Leonardo F.R. Ribeiro, Zimeng Qiu, Markus Dreyer, Akari Asai, Chenyan Xiong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05495

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deep Research Comparator: A Platform For Fine-grained Human Annotations of Deep Research Agents Prahaladh Chandrahasan, Jiahe Jin, Zhihan Zhang, Tevin Wang, Andy Tang, Lucy Mo, Morteza Ziyadi, Leonardo F.R. Ribeiro, Zimeng Qiu, Markus Dreyer, Akari Asai, Chenyan Xiong Abstract Effectively evaluating deep research agents that autonomously search the web, analyze information, and generate reports remains a major challenge, particularly when it comes to assessing long reports and giving detailed feedback on their intermediate steps. To address these gaps, we introduce Deep Research Comparator , a platform that offers a holistic framework for deep research agent hosting, side-by-side comparison, fine-grained human feedback collection, and ranking calculation. Given a user query, our platform displays the final reports from two different agents along with their intermediate steps during generation. Annotators can evaluate the overall quality of final reports based on side-by-side comparison, and also provide detailed feedback separately by assessing intermediate steps or specific text spans within the final report. Furthermore, we develop Simple Deepresearch , an end-to-end agent scaffold. This scaffold serves as a baseline that facilitates the easy integration of various large language models to transform them into deep research agents for evaluation. To demonstrate the platform’s utility for deep research agent development, we have collected real user preference data from 17 annotators on three deep research agents. We plan to release the code base and collected data. 1 1 1 Our code is available at GitHub , and demo video can be found at Youtube . 1 1 footnotetext: Equal Contribution. 1 Introduction Recent advancements in large language models (LLMs) have catalyzed the development of autonomous agents, which can utilize various tools to solve real-world tasks. Among them, the deep research agent has recently emerged, referring to an agent powered by LLMs that autonomously finds, analyzes, and synthesizes vast amounts of online information to produce a comprehensive report Xu and Peng 2025 . Prominent proprietary examples include agents developed by OpenAI OpenAI 2025 , Google Gemini Gemini 2025 , and Perplexity perplexity 2025 , and the open-source community has also produced notable contributions Li et al. 2025 ; GPTResearcher 2025 . Figure 1: The main interface of Deep Research Comparator . Users can submit their query in the text box, and the intermediate steps and final reports from two agents will be displayed side by side. Despite these advancements, evaluating the long-form reports generated by deep research agents remains a significant challenge. Existing benchmarks for search agents Wei et al. 2025 ; Wu et al. 2025 primarily target short-form answers, leaving the assessment of long-form reports largely underexplored. Moreover, unlike tasks such as math or coding, there are no widely accepted objective metrics for assessing long reports ( Asai et al. 2024 ) . To this end, some existing work focuses on developing hand-crafted rubrics for evaluation Gou et al. 2025 ; FutureSearch et al. 2025 ; Arora et al. 2025 , but these approaches are difficult to scale. Beyond the difficulty in assessing long reports, we also lack a fine-grained evaluation of agents’ intermediate steps. LLM-based agents often execute multi-step actions to complete long-horizon tasks, but current evaluations are typically based on outcomes Deng et al. 2023 ; Zhou et al. 2024 ; Jimenez et al. 2024 ; Xie et al. 2024 , resulting in sparse and delayed feedback. However, process feedback is crucial for training methods like reinforcement learning (RL) to accurately attribute step contribution in the trajectory Feng et al. 2025 ; Wang et al. 2025 ; Setlur et al. 2025 . Such feedback is also essential for understanding agent behavior and assessing the reliability of the generated reports Lightman et al. 2023 ; Wan et al. 2024 . To address these gaps, we introduce Deep Research Comparator (Section 3 ), a platform for fine-grained human evaluation of long-report generation by deep research agents. It supports easy integration of new agents with minimal engineering effort, enabling both open-source and closed-source systems to be hosted through a unified web interface. When a user submits a query, the platform randomly routes it to two agents and presents their intermediate steps and final reports. Deep Research Comparator enables the collection of two types of human feedback. First, users can perform a side-by-side comparison of the two final reports and vote for the better one (Figure 1 bottom), similar to Chatbot Arena Chiang et al. 2024 . These preference votes can then be used to calculate an outcome-based ranking of the agents. Second, users can provide more fine-grained annotations by upvoting or downvoting individual intermediate steps and specific text spans within the reports to indicate their quality (Figure 2 ). This detailed feedback creates a valuable dataset for potential future use, such as agent fine-tuning or reward modeling. To support easy deployment and evaluation of different LLMs as deep research agents, we developed Simple Deepresearch , an end-to-end agent scaffold serving as a baseline (Section 4 ). This prompt-based scaffold allows various LLMs to be easily integrated, transforming them into deep research agents for evaluation. Using our platform, we conducted a proof-of-concept experiment on three agents with 176 user queries (Section 5 ). We then ranked the agents based on pairwise comparison votes on their final reports. Additionally, we collected 1,281 upvote/downvote feedback on intermediate steps and 593 upvote/downvote with highlighted text spans in reports. From this, we calculated each agent’s overall upvote rate on intermediate steps for a process-based evaluation metric. Based on the collected data, we derive some design considerations for future agent developers on how to effectively use our platform for annotation and evaluation. Overall, we make three key contributions: 1. We provide Deep Research Comparator , an open source platform to host deep research agents and supports fine-grained human evaluation through a unified web interface. 2. We develop Simple Deepresearch , a baseline agent scaffold that allows for the easy integration of various LLMs into deep research agents for evaluation. 3. We demonstrate the utility of our platform by collecting 176 real-world user queries with comparison votes, and fine-grained annotation on 1,281 steps and 593 text spans in reports. 2 Related Work Figure 2: Demonstration of fine-grained annotation. Left: Users can upvote/downvote each step in the generation process. Right: Users can highlight any text span in the final report and upvote/downvote it. LLM-based search agents Search Agents are LLM-based systems that autonomously and iteratively use search tools to gather external information Jin et al. 2025 ; Schick et al. 2023 ; Anthropic 2025 ; Chen et al. 2024 for solving complex search tasks. These agents can generate dynamic action decisions that are executed as real-time internet searches, and then synthesize this information to provide up-to-date and relevant responses. Among them, “deep research agents” represent a more capable evolution of this paradigm OpenAI 2025 ; GPTResearcher 2025 ; Li et al. 2025 , which conduct substantially more searches, engage in long-horizon planning, and operate with significantly higher test-time compute budgets, for more demanding, research-intensive scenarios. Evaluations for Search Agents Evaluations for search agents can be roughly divided into two types: short-answer tasks and long-form report generation. Short-answer tasks are easier to evaluate due to the availability of definitive ground truth. Commonly employed evaluations include intricate tasks that demand

Read the complete text