Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning
Banghao Chi, Yining Xie, Mingyuan Wu, Jingcheng Yang, Jize Jiang, Zhaoheng Li, Shengyi Qian, Minjia Zhang, Klara Nahrstedt, Rui Hou, Xiangjun Fan, Hanchao Yu · 2026
Read Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning by Banghao Chi, Yining Xie, Mingyuan Wu, Jingcheng Yang, Jize Jiang, Zhaoheng Li, Shengyi Qian, Minjia Zhang, Klara Nahrstedt, Rui Hou, Xiangjun Fan, Hanchao Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22642.txt.
Introduction
Chi and others train spreadsheet agents by reinforcement learning inside a realistic Excel setting, replacing prompt only agents that falter on long multi step workflows. Their Spreadsheet-RL system gathers paired start and goal sheets from online forums and adds a finance and supply chain benchmark. Training runs inside a Spreadsheet Gym exposing Excel tools through a Python sandbox. Tests lift a Qwen3 4B model from 12.0 percent to 23.4 percent on SpreadsheetBench and from 8.4 percent to 17.2 percent on the new domain set.
Catalog record
| CREATOR | Banghao Chi, Yining Xie, Mingyuan Wu, Jingcheng Yang, Jize Jiang, Zhaoheng Li, Shengyi Qian, Minjia Zhang, Klara Nahrstedt, Rui Hou, Xiangjun Fan, Hanchao Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64754 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22642
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning Banghao Chi, Yining Xie, Mingyuan Wu, Jingcheng Yang, Jize Jiang, Zhaoheng Li, Shengyi Qian, Minjia Zhang, Klara Nahrstedt, Rui Hou, Xiangjun Fan, Hanchao Yu Abstract Spreadsheet systems (e.g., Microsoft Excel, Google Sheets) play a central role in modern data-centric workflows. As AI agents grow increasingly capable of automating complex tasks, such as controlling computers and generating presentations, building an AI-driven spreadsheet agent has emerged as a promising research direction. Most existing spreadsheet agents rely on specialized prompting over general-purpose LLMs; while this design has potentials on simple spreadsheet operations, it struggles to manage the complex, multi-step workflows typical of real-world applications. In this paper, we introduce Spreadsheet-RL , a reinforcement learning (RL) fine-tuning framework designed to train specialized spreadsheet agents within a realistic Microsoft Excel environment. Spreadsheet-RL features an automated pipeline for scalable collection of paired start-goal spreadsheets from online forums, as well as domain-specific evaluation tasks in areas such as finance and supply chain management, which we compile into the new Domain-Spreadsheet benchmark dataset. It also includes a Spreadsheet Gym environment designed for multi-turn RL: Spreadsheet Gym exposes extensive Excel functionality through a Python sandbox, along with a refined harness that incorporates a comprehensive tool set and carefully designed tool-routing rules for spreadsheet tasks. Through comprehensive experiments, we show that Spreadsheet-RL substantially enhances AI agent’s performance on both general and domain-specific spreadsheet tasks: it improves Qwen3-4B-Thinking-2507’s Pass@1 on SpreadsheetBench from 12.0% to 23.4% , and raises Pass@1 from 8.4% to 17.2% on our curated Domain-Spreadsheet dataset. These results highlight Spreadsheet-RL ’s strong potential for generalization and real-world adoption in spreadsheet automation, and broadly, its promise for advancing LLM-based interactions with data interfaces in everyday work. We will release the training data, environment, and training pipeline to facilitate future research on spreadsheet agents. † † footnotetext: All data and code releases are maintained by the corresponding authors at UIUC and are not affiliated with Meta. † † footnotetext: * Contributed equally to this work, where more junior authors are listed ahead of senior. † † footnotetext: † Project Lead. https://huggingface.co/datasets/Spreadsheet-RL/Spreadsheet-RL https://github.com/Spreadsheet-RL/Spreadsheet-RL 1 Introduction Spreadsheet systems, such as Microsoft Excel, Google Sheets, WPS Sheets, and LibreOffice, are widely adopted in data-centric workflows [ 7 , 27 ] . They support tasks from personal activities such as travel planning and household budgeting, to professional duties such as financial modeling and data presentation [ 27 , 19 , 14 ] . As AI agents grow in both popularity and capability for automating tasks traditionally performed by humans, such as computer use and slide deck design [ 33 , 15 , 8 , 37 ] , the development of an AI agent for spreadsheets (a spreadsheet agent ) to automate the human-operated spreadsheet-centered workflows will have the potential to fundamentally reshape how data science is performed at scale. Qwen3-4B Instruct Qwen3-14B Qwen3-8B Qwen3-32B GPT-4o OpenAI o3 Qwen3-4B Thinking Base + Native Harness + Full Tools + RL Training 0 5 10 15 20 25 Benefits of Spreadsheet-RL Model / stage Accuracy Score Figure 1 : The rightmost four highlighted bars trace the main Qwen3-4B-Thinking-2507 result sequence: raw base model, spreadsheet-native interaction harness, comprehensive spreadsheet-tool access, and Spreadsheet-RL post-training. Representative open-source and closed-sourced baselines are provided for scale reference. While there exists recent research such as SheetCopilot [ 17 ] , SheetAgent [ 5 ] , and ChatGPT Agent [ 23 ] studying spreadsheet agents, their approaches notably rely on proprietary (and powerful) Large Language Models (LLMs) with reasoning such as GPT-4o [ 13 ] , which have sufficient general capabilities to perform simple spreadsheet operations via natural language instructions. That is, these works are limited in that they rely on advancements in general LLMs, and their prompting strategies, rather than specific improvements in how the LLM agents utilize spreadsheets. This limitation leads to existing spreadsheet agents struggling to reliably execute more complex, multi-step workflows that dominate real-world spreadsheet use; for example, the ChatGPT Agent and Copilot (both with excel access) [ 23 ] , reach 45.5% and 20.0%, respectively, on SpreadsheetBench [ 18 ] . On the other hand, frontier industry labs have recently begun to develop specialized spreadsheet agents, yet adopt undisclosed approaches that rely on internal benchmarks and closed training pipelines [ 24 , 20 , 9 ] . One promising approach a powerful, specialized, and open-source spreadsheet agent can be built is reinforcement learning (RL) fine-tuning: following DeepSeek-R1 [ 11 ] , on-policy RL combined with rule-based, verifiable outcome rewards has improved mathematical reasoning [ 28 , 32 ] , visual reasoning [ 36 ] , and enabled scalable post-training for agentic domains such as software engineering [ 34 , 35 ] , web interaction [ 2 ] , data [ 30 , 22 ] , and computer use [ 16 , 37 , 31 ] . However, applying the same approach to spreadsheets is challenging: Unlike many web or software tasks where success can be validated by unit tests or binary completion signals, final spreadsheets are produced by a long sequence of operations involving values, formulas, and layout. This leads to significant difficulties: ① collecting sufficient initial–final spreadsheet pairs for training is expensive and difficult to scale for RL training; and ② without step-by-step supervised fine-tuning data, which is even more costly to obtain, the agent must begin RL from a weak interaction policy. This makes a spreadsheet-specific harness essential for providing a structured action space and workflow prior that enable a meaningful initial success rate. In this paper, we introduce Spreadsheet-RL , a framework for building specialized spreadsheet agents that, to the best of our knowledge, features the first end-to-end RL post-training method for the spreadsheet domain. Spreadsheet-RL differs from prior prompt-driven works by utilizing on-policy RL (e.g., GRPO [ 28 ] ) in a real-world spreadsheet environment. First, for training data collection, Spreadsheet-RL features the automated Spreadsheet Data Agent that collects and constructs large-scale realistic spreadsheet tasks for outcome-based rewards across specialized domains such as finance, human resources, and supply chain management; next, for performing operations, the Spreadsheet Gym —a multi-turn interactive Microsoft Excel environment integrated with a code sandbox, supports a broad range of advanced Excel functionalities. Finally, Spreadsheet-RL combines the components into a purpose-built asynchronous RL training framework that interfaces seamlessly with long-horizon, multi-turn spreadsheet interactions, supported by a carefully designed agent harness that incorporates a comprehensive tool set, refined tool-routing rules, and workflow for spreadsheet tasks. We apply Spreadsheet-RL to the Qwen3 series Large Language Models [ 38 ] with GRPO [ 11 ] objectives to build specialized spreadsheet agents for evaluation on ① SpreadsheetBench [ 18 ] , the largest open-source benchmark, and ② Domain-Spreadsheet , the first open-source domain-specific spreadsheet benchmark we curate. On SpreadsheetBench, Spreadsheet-RL improves Qwen3-4B-Thinking-2507 from 12.0% to 23.4% Pass@1. These gains, summarized in Figure 1 , show how spreadsheet-native harness design, richer tool access, and RL post-training each improve the same 4B open-source base model. Our results also demonstrate that Spreadsheet-RL generalizes across real-world spreadsheet tasks from specialized domains: on Domain-Spreadsheet, Spreadsheet-RL improves overall pass@1 from 8.4% to 17.2% (Table 2 ). Finally, training dynamics and qualitative analysis show that RL improves not only final accuracy