Benchmarking Real-Time Question Answering via Executable Code Workflows

Wenjie Zhou, Yuan Gao, Xin Zhou, Hao Fu, Zhongjian Miao, Wei Chen, Bo Chen, Xiaobing Zhao · 2026

Read Benchmarking Real-Time Question Answering via Executable Code Workflows by Wenjie Zhou, Yuan Gao, Xin Zhou, Hao Fu, Zhongjian Miao, Wei Chen, Bo Chen, Xiaobing Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16349.txt.

Introduction

Zhou and seven co-authors present RT-QA, a benchmark for real-time question answering via code workflows. Existing benchmarks are mostly static, missing the temporal dynamics of information and the evolving nature of real-world knowledge. RT-QA uses an agent-driven pipeline. Its code crawls the web and extracts answers from DOM structure, producing real-time ground truth, with a self-repair mechanism adapting to web page structures. Spanning 12 domains with 320 Chinese questions across three difficulty levels, it reveals limits in real-time adaptability, even for top models.

Catalog record

CREATORWenjie Zhou, Yuan Gao, Xin Zhou, Hao Fu, Zhongjian Miao, Wei Chen, Bo Chen, Xiaobing Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59278
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16349

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking Real-Time Question Answering via Executable Code Workflows Wenjie Zhou, Yuan Gao, Xin Zhou, Hao Fu, Zhongjian Miao, Wei Chen, Bo Chen, Xiaobing Zhao Abstract Retrieving real-time information is a fundamental capability for search-integrated agents in real-world applications. However, existing benchmarks are predominantly static and therefore fail to capture the temporal dynamics of information and the continuously evolving nature of real-world knowledge. To address this limitation, we propose RT-QA , a dynamic evaluation framework that leverages executable code workflows to retrieve up-to-date answers at evaluation time. Specifically, we construct an agent-driven pipeline that autonomously generates code for web crawling and DOM-based answer extraction to produce real-time ground truth. To ensure robust evaluation over time, the pipeline further incorporates a self-repair mechanism to adapt to changes in web page structures. RT-QA spans 12 domains (e.g., Finance, Sports) with 320 Chinese questions categorized into three difficulty levels. Extensive evaluations of state-of-the-art models (e.g., GPT-5.2, GLM-4.7) reveal significant limitations in real-time adaptability: even the best models achieve only ∼ \sim 46% accuracy. Our analysis highlights two primary failure modes: (1) Lazy Retrieval , where agents rely on search snippets instead of deeply scanning specific websites for information (20% of failures); and (2) Temporal Confusion , a cognitive error where agents retrieve a historical date (e.g., an event in 2024) and fail to re-anchor to the current time (2026) for subsequent reasoning. These findings suggest that future agents require not just better retrieval strategies, but robust temporal state management. The code and data are available at https://github.com/leaves-slient/RT-Bench.git . 1 Introduction Figure 1: Comparison between Static QA and Real-time QA. Unlike static questions (top) where answers are fixed or can be pre-stored, real-time questions (bottom) typically involve relative temporal constraints (e.g., “yesterday” , “past 7 days” ). The capability to access and reason over real-time information is a fundamental requirement for general intelligence. In recent years, Large Language Models (LLMs) have evolved from static knowledge bases into dynamic, search-integrated agents. The emergence of commercial systems like SearchGPT 14 , Perplexity 17 , and the DeepResearch framework 20 highlights this shift, demonstrating impressive performance in complex information-seeking tasks 13 ; 19 and effectively transitioning LLMs into comprehensive assistants. Despite this rapid development, evaluation methodologies for such agents remain limited. The majority of existing benchmarks rely on static question-answer pairs 18 ; 9 ; 26 , suffering from two critical flaws: Data Leakage and Knowledge Obsolescence . The former enables agents to answer from memory instead of conducting necessary searches, while the latter renders the evaluation ground truth invalid as facts evolve. To mitigate this, recent studies have introduced periodic benchmarks that update monthly or annually 11 ; 23 ; 8 . While some works propose automated pipelines 3 ; 2 , they remain limited by their dependency on pre-existing structured sources (e.g., Wikidata or specific APIs). Ideally, answering daily questions—such as tracking stock prices or arXiv submissions—requires querying official APIs. But such APIs are often scarce, paywalled, or simply do not exist. Consequently, existing benchmarks must compromise, covering only unchanged facts or a few API-supported domains. This leaves a critical gap: agents are rarely tested on high-frequency information from the open web, where no direct API exists. In this paper, we introduce RT-QA , a dynamic benchmark designed to evaluate agents on rapidly changing, daily-updated information. The core innovation of RT-QA is the paradigm shift from Static Answers to Executable Code Workflows . Instead of storing a fixed answer string, each test item in RT-QA consists of a natural language question paired with a Python-based workflow. This workflow functions by visiting authoritative websites, parsing unstructured DOM elements, and generating the real-time ground truth at the exact moment of evaluation. This paradigm allows us to construct high-value questions across 12 domains (including Finance, Public Policy, and Science) that update daily, weekly, or monthly, minimizing manual maintenance. To realize this approach, we propose the RT Agent framework, which leverages the tool-creation capabilities of LLMs to automatically generate these workflows and includes a self-repair mechanism to handle inevitable changes in web page structures. We conducted extensive evaluations of state-of-the-art agents, including GPT-5.2, Claude-4.5, and GLM-4.7 21 , on RT-QA. The results reveal a significant gap: while these models excel on static tasks, they struggle with real-time inquiries. Even the leading models achieve an average accuracy of only 46%, with many other models falling below 40%. To understand the root causes, we conducted a large scale analysis of over 7,000 agent trajectories. Our taxonomy of failure modes reveals that the primary bottleneck remains information gathering : Retrieval-related issues (including Lazy Retrieval, Incomplete Scan, and Source Quality) account for 45% of all errors. Specifically, 20% of failures stem from Lazy Retrieval, where agents rely on search snippets without visiting the actual page. Moreover, we uncover a recurring reasoning error: Temporal Confusion. We observe that agents frequently perform worse on intermediate reasoning tasks (Level 2) than on highly complex ones (Level 3). Our analysis suggests this stems from poor temporal state management—agents mistakenly use historical dates retrieved in intermediate steps as the “current time” for subsequent reasoning, whereas the higher complexity of L3 tasks forces a more rigorous isolation of temporal variables. Our contributions are summarized as follows: • We identify the lack of methods for evaluating real-time information processing, proposing RT-QA , the first benchmark using executable workflows to dynamically generate ground truth. • We develop the RT Agent , an automated framework for generating and maintaining these workflows, allowing users to easily extend the benchmark. • We provide a comprehensive evaluation and a fine-grained failure taxonomy based on over 7,000 trajectories. We highlight Lazy Retrieval and Temporal Confusion as the two dominant failure modes, offering critical insights for future agents to improve retrieval depth and temporal state management. Figure 2: An Example of an RT-QA Executed Workflow. The figure demonstrates how a single Python workflow handles dynamic time anchoring. The code (left) explicitly calculates the target date (tomorrow) based on the execution time. Consequently, executing the same workflow on different dates (Oct 31 vs. Nov 1) yields correct, context-aware answers (bottom right), effectively functioning as a dynamic pseudo-API. 2 Related Work 2.1 From Static RAG to Real-Time Agents While LLMs excel at reasoning, their internal knowledge is static. To mitigate this Knowledge Obsolescence, Retrieval-Augmented Generation (RAG) methods 10 ; 6 connect models to external corpora. Recent works like TimeRAG 25 and Time-Aware LMs 5 further improved temporal grounding. However, standard RAG relies on pre-indexed documents, which inevitably suffer from indexing latency—they cannot answer questions about events that happened minutes ago. To overcome this, the field is shifting towards autonomous agents that interact directly with the live web. Early works like WebGPT 13 and WebArena 29 demonstrated how agents can use browsers as tools. More recently, advanced frameworks like DeepResearch 20 have enabled agents to perform concurrent browsing and complex planning. Our work builds on this trend but focuses on evaluation: providing a rigorous benchmark to measure how well these agents actually perform in dynamic, real-world scenarios. 2.2 Benchmarks for Time-Sensitive QA Evaluating temporal reasoning has evolved significantly. Early datasets like TempQuestions 7 and Situational QA 27 focused on static text. Recent dynamic benchmarks have pushed the boundary: StreamingQA 11 and

Read the complete text