ORBIT : Scalable and Verifiable Data Generation for Search Agents on a Tight Budget

Nandan Thakur, Zijian Chen, Xueguang Ma & Jimmy Lin · 2026

Read ORBIT : Scalable and Verifiable Data Generation for Search Agents on a Tight Budget by Nandan Thakur, Zijian Chen, Xueguang Ma & Jimmy Lin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01195.txt.

Introduction

Nandan Thakur and co-authors present ORBIT in 2026, a system for training search agents. Search agents join language models with web search for hard questions. Building training data for deep research tasks remains costly and hard.

Such tasks involve multi-step retrieval and reasoning over many sources. Human annotation is expensive, and other routes carry cumbersome prerequisites, they note. ORBIT aims at scalable and verifiable data generation on a tight budget. Its title promises exactly that economy with checks on quality. Cheaper, verified training data could widen who can build research agents. The paper speaks to teams training agents without large annotation budgets. It treats data generation itself as the bottleneck to solve.

Catalog record

CREATORNandan Thakur, Zijian Chen, Xueguang Ma & Jimmy Lin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604-01195

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ORBIT : Scalable and Verifiable Data Generation for Search Agents on a Tight Budget Nandan Thakur, Zijian Chen, Xueguang Ma & Jimmy Lin Abstract Search agents, which integrate language models (LMs) with web search, are becoming crucial for answering complex user queries. Constructing training datasets for deep research tasks, involving multi-step retrieval and reasoning, remains challenging due to expensive human annotation, or cumbersome prerequisites. In this work, we introduce ORBIT , a training dataset with 20K reasoning-intensive queries with short verifiable answers, generated using a frugal framework without relying on paid API services. The modular framework relies on four stages: seed creation, question–answer pair generation, and two stages of verification: self and external. ORBIT spans 15 domains and each training pair requires 4–5 reasoning steps, with external search verification required from the complete web. We train Qwen3-4B as the base model on ORBIT using GRPO and evaluate it on Wikipedia question answering tasks. Extensive experiment results demonstrate that ORBIT-4B achieves strong performance among sub-4B LLMs as search agents, proving the utility of synthetic datasets. Our framework, code and datasets are open-sourced and available publicly. castorini/orbit orbit-ai 1 Introduction Large language models (LLMs) have been used for their reasoning capabilities beyond simple factoid queries in a new frontier we refer to as deep search . This setting requires interleaved reasoning, decomposing complex tasks, and reasoning across multiple sources of information, and it is now exposed as an endpoint in multiple commercial products ( OpenAI 2025 ; Gemini 2025 ) . Unlike traditional question-answering systems, answering deep search questions goes beyond a single-turn web search: it requires breaking down complex queries, conducting multiple retrieval steps, and iteratively planning searches and aggregating results from a search tool. Recent work training search agents, such as Search-R1 and InfoSeek ( Jin et al. 2025 ; Xia et al. 2026 ) has shown that applying GRPO with verifiable rewards improves models’ iterative reasoning and search tool use. Training data availability for deep search tasks is very limited. Search-R1 trains on NQ ( Kwiatkowski et al. 2019 ) and HotpotQA ( Yang et al. 2018 ) datasets; however, these contain simple retrieval queries that do not necessarily require multiple hops of searches for answering them. Collecting human annotations with verifiable answers for complex queries is also challenging: given a complex query, the uncertainty of web search makes the annotation burden impractically high. On the other hand, training datasets with complex search queries are either limited in training pairs, or require pre-requisites before generation, such as knowledge-graph construction, limiting their real-world applicability and usage ( Wu et al. 2025 ; Sun et al. 2025b ; Xi et al. 2025 ; Tao et al. 2026 ; Liu et al. 2025 ; Wolfson et al. 2026 ; Xia et al. 2026 ) . In this work, we introduce a frugal framework for constructing complex training data for search agents in a budget constrained manner, without relying on any paid API services. The framework is modular and can be adopted easily for synthetic data generation. Existing synthetic deep search datasets commonly rely on graph structures and entity linkages, or are not fully open sourced, as shown in Table 1 . Using our framework, we construct ORBIT ( Open-Web Reasoning for Information Retrieval Tasks ), a verified synthetic training dataset with narrative-style, i.e., long and reasoning-intensive questions with short verifiable answers, requiring multi-hops across the web. Specifically, ORBIT is constructed by a four-stage framework that automatically constructs verifiable, multi-hop training pairs at scale: (1) seed creation : we expand 15 domains into large sets of Wikipedia categories and iterate linked pages, using page titles as seeds to ensure both head and long-tail coverage; (2) question–answer pair generation : conditioned on a seed, a search-enabled DeepSeek-V3.1 composes an inverted question paired with a short verifiable answer; (3) self-verification : a search-enabled DeepSeek-V3.1 assesses whether the proposed answer satisfies every atomic fact mentioned in the inverted question and provides supporting URLs as citations, discarding pairs with unverified answers or weakly supported sub-claims; and (4) external-verification : two cascading LLM judges ( Qwen3-4B-Instruct-2507 and gpt-oss-120b ) with scraped URL context, generate an answer and self-verify the generated answer as the judge with the ground-truth answer. Using ORBIT, we fine-tune smaller and efficient LLMs (<4B parameters) as search agents using Qwen3-4B as the base model ( Yang et al. 2025a ) , with GRPO ( Shao et al. 2024 ) . By mixing ORBIT with training datasets, ORBIT-4B demonstrates strong effectiveness over sub-4B search agent baselines on complex Wikipedia-based QA benchmarks, highlighting the potential of our framework for data generation to produce high-quality supervision for scaling small-sized search agents. We summarize our major contributions as follows: 1. We propose a frugal construction framework without requiring any pre-requisites or relying on expensive API’s, to generate reasoning-intensive queries with short and verifiable answers on a tight budget. 2. We introduce ORBIT, one of the largest available datasets for training search agents with 20K+ verified question–answer pairs with a hard complexity, requiring verification from both Wikipedia and Web domains for each question. 3. We empirically demonstrate the effectiveness of ORBIT through extensive experiments, showing that the proposed dataset yields significant performance gains on small-sized search agents. ORBIT-4B surpasses existing search agents with less than 4B model parameters by up to 9.0 EM accuracy on complex QA benchmarks on Wikipedia. 4. We open-source the framework, ORBIT dataset and the code for training search agents. Code is available on GitHub at: https://github.com/castorini/orbit , ORBIT dataset and search agents are available on Hugging Face at: https://huggingface.co/orbit-ai . Dataset Name Source Train Pairs Complexity Pre-requisite / Constraint NQ ( Kwiatkowski et al. 2019 ) Wiki 300K+ Easy None HotpotQA ( Yang et al. 2018 ) Wiki 100K+ Easy None WebWalkerQA ( Wu et al. 2025 ) Web 14.3K Easy Web Traversal SimpleDeepS. ( Sun et al. 2025b ) Wiki & Web 871 Easy Web Search & HTML Extract InfoDeepSeek ( Xi et al. 2025 ) Web 245 Easy Human Annotators WebShaper ( Tao et al. 2026 ) Wiki 500 Hard Wikipedia Hyperlinks WebExplorer ( Liu et al. 2025 ) Web 100 Hard Web Search & Obfuscation MoNaCo ( Wolfson et al. 2026 ) Wiki 1,315 Hard Human Annotators DeepDive ( Lu et al. 2025 ) KG 3,250 Hard Knowledge Graph InfoSeek ( Xia et al. 2026 ) Wiki & Web † 17.8K Hard Entity Linking & Obfuscation ORBIT (ours) Wiki & Web 20K Hard None Table 1: Comparison of ORBIT against other training datasets for search agents. Prior datasets require prior constraints such as a knowledge graph (KG), lack structural depth or remain limited in scale. ORBIT is a large-scale multi-hop dataset, requiring no prior constraints and containing complex queries, that can be scaled easily with a low budget. 2 Related Work 2.1 Reinforcement Learning and Search-Augmented LLMs Reinforcement learning (RL) improves agent performance by learning from previous experience and maximizing cumulative rewards ( Sutton & Barto 1998 ) . Recently, RL with group-based methods such as GRPO ( Shao et al. 2024 ) have enabled LLMs to solve complex tasks such as Olympiad-level math problems as well as broader reasoning tasks ( Shao et al. 2024 ; Yu et al. 2025 ;

Read the complete text