GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git

Tobias Lindenbauer, Egor Bogomolov, Yaroslav Zharov · 2025

Read GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git by Tobias Lindenbauer, Egor Bogomolov, Yaroslav Zharov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22583.txt.

Introduction

Tobias Lindenbauer, Egor Bogomolov, and Yaroslav Zharov build, in a 2025 benchmark paper, a test suite for how well AI agents handle Git. Git is the version control tool at the heart of software work. Coding benchmarks judge code writing but skip the daily chores of branching, merging, and history care. Their suite draws real tasks from open Python, Java, and Kotlin projects.

The full set holds nine hundred graded tasks. It also offers a small rapid version for quick trials and a large training pool. A strong commercial model with custom tools solved about a fifth of the rapid tasks. The authors offer the benchmark as a step toward agents that can do the whole developer job.

Catalog record

CREATORTobias Lindenbauer, Egor Bogomolov, Yaroslav Zharov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22583

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GitGoodBench: A Novel Benchmark For Evaluating Agentic Performance On Git Tobias Lindenbauer, Egor Bogomolov, Yaroslav Zharov Abstract Benchmarks for swe ( swe ) AI agents, most notably SWE-bench, have catalyzed progress in programming capabilities of AI agents. However, they overlook critical developer workflows such as vcs ( vcs ) operations. To address this issue, we present GitGoodBench 1 1 1 https://github.com/JetBrains-Research/git-good-bench , a novel benchmark for evaluating AI agent performance on vcs ( vcs ) tasks. GitGoodBench covers three core Git scenarios extracted from permissive open-source Python, Java, and Kotlin repositories. Our benchmark provides three datasets: a comprehensive evaluation suite (900 samples), a rapid prototyping version (120 samples), and a training corpus (17,469 samples). We establish baseline performance on the prototyping version of our benchmark using GPT-4o equipped with custom tools, achieving a 21.11% solve rate overall. We expect GitGoodBench to serve as a crucial stepping stone toward truly comprehensive swe agents that go beyond mere programming. LLM Large Language Model SLM Small Language Model CoT Chain-of-Thought SE Software Engineering VCS Version Control System FCC File-Commit Chain EM Exact-Match IR Interactive Rebase ICC Iterative Committing of Changes MCR Merge Conflict Resolution PL Programming Language MCP Model-Context Protocol (a) Merge Conflict Resolution: The agent must reproduce the ground-truth merge commit given a set of conflicts. (b) Interactive Rebase: The agent generates an alternative history based on existing commits. (c) Iterative Committing of Changes: The agent generates an alternative based on a disorganized set of changes. We only use the original commit history for evaluation. Figure 1: The three Git scenarios supported by GitGoodBench. Each scenario benchmarks a typical Git use-case and unique aspect of version control. 1 Introduction While the rapid scaling of llm has led to promising results across various tasks initially, the improvements gained from scaling models further are slowing down. Compared to GPT-3 Brown et al. (2020) , GPT-3.5 achieves a approximately 60\% improvement OpenAI et al. (2024a) on MMLU Hendrycks et al. (2021) . The improvement from GPT-3.5 to GPT-4, however, is just approximately 23\% OpenAI et al. (2024a) . Scaling test-time compute rather than just models has emerged as an alternative for further improving performance, leading to the rise of AI agents Yao et al. (2023) ; Shinn et al. (2023) ; Wang et al. (2024) . AI agents equip llm with external tools Schick et al. (2023) and employ sophisticated planning and reasoning strategies such as ReAct Yao et al. (2023) or Reflexion Shinn et al. (2023) to dynamically adjust in uncertain environments. swe ( swe ) emerged as a pivotal application domain due to the availability of high-quality data in open-source repositories and because the creation and maintenance of software underpins innovation and economic impact across virtually every sector. SWE-bench Jimenez et al. (2024) is the industry-standard benchmark for evaluating the agent’s programming proficiency through testing the agent’s ability to fix bugs in real-world software. This spurred the rapid development of AI agents for programming by major players in the tech tooling ecosystem Cursor (2024) ; Basu et al. (2024) ; Zakonov (2025) ; Microsoft (2025) ; Anthropic (2025) . vcs ( vcs ), such as Git, are ubiquitous in swe Cortés Ríos et al. (2022) and play a pivotal role in building software in distributed teams. It is thus natural to use Git as a medium of collaboration between AI agents and human engineers. While llm providers are advertising the Git capabilities of their systems Anthropic (2025) , there currently exists no benchmark for evaluating an AI agent’s capacity of interacting with Git in an end-to-end manner. Furthermore, typical Git tasks such as ir ( ir ) are time-consuming and distinct from raw code-generation. ir requires reasoning over the Git history and an in-depth understanding of dependencies between the commits constituting the history. To stimulate innovation in the direction of comprehensive, end-to-end swe AI agents that go beyond mere programming, we introduce a novel benchmark for the popular vcs Git. This comprises a training corpus for collecting agentic trajectories and two evaluation sets (lite and full). The benchmark supports mcr ( mcr ), ir ( ir ), and the icc ( icc ) ( Figure 1 ). We scrape all data from permissive, open-source, Python, Java, or Kotlin GitHub repositories. Furthermore, we provide a baseline implementation using GPT-4o OpenAI et al. (2024b) with custom tools, achieving a 21.11% solve rate. 2 Related Work Several benchmarks, such as SWE-bench Jimenez et al. (2024) , or the Kowinski prize Konwinski et al. (2024) evaluate agentic systems on complex, multi-turn swe tasks sourced from real-world GitHub issues. While the environment allows Git usage, the evaluation focuses solely on whether the agent resolves the bug rather than how it leverages VCS. In contrast, our benchmark explicitly measures an agent’s proficiency with Git tasks. This allows future research to thoroughly examine and refine vcs -focused strategies in swe agents and tailor agents to vcs tasks specifically. While previous works on automating or evaluating mcr Svyatkovskiy et al. (2022) ; Shen et al. (2023) ; Boll et al. (2024) ; Pan et al. (2021) and commit message generation or completion Jiang et al. (2017) ; Hal et al. (2019) ; Eliseeva et al. (2023) exist, they exclusively cater to specific vcs subtasks. In contrast, our benchmark is the first to encapsulate multiple subtasks, such as commit message generation, reasoning across commits, and rebase plan generation into a single benchmarking scenario. This uniquely positions GitGoodBench for evaluating and training AI agents with expertise in vcs tasks in end-to-end settings. Filter Reason Repository size \leq 400 MB Avoid Git LFS heavy repositories Repository stars \geq 1000 Heuristic for repository quality Repository is not archived Heuristic for repository quality Repository is not forked Avoid duplication Last commit within a month of May, 31st 2024 Avoid stale repositories Repository has permissive license Ensure legal compliance Repository \geq 5 branches Heuristic for merge conflict scenarios Repository \geq 5 contributors Heuristic for merge conflict scenarios (a) Repository metadata filters we use for selecting the initial repositories we consider in the benchmark creation. We consider the following licenses permissive: MIT, Apache 2.0, BSD 3-Clause “New” or “Revised”, BSD 2-Clause “Simplified”. Filter Reason No merge commit in fcc ( fcc ) Cleanly separate scenario types No merge conflict in unsupported file type Only support Python, Java, or Kotlin All merge scenarios contain conflict Merges without a conflict are trivial Merge scenarios have exactly two parents Avoid dilution by complex and rare merge types Number of merge conflicts \leq 8 Ensure the agent can theoretically solve the scenario Length of fcc \leq 6 Ensure the agent can theoretically solve the scenario fcc file is modified, not added Otherwise we get a single hunk when resetting (b) Scenario level filters for selecting scenarios to include in our benchmark. Table 1: Filters for selecting repositories and scenarios to include in our benchmark. 3 GitGoodBench Datasets We provide GitGoodBench (900 samples) and GitGoodBench Lite (120 samples) for evaluation in comprehensive and rapid-prototyping settings, respectively. The research community recently started investigating swe agents powered by fine-tuned slm Pan et al. (2024) ; Jain et al. (2025) ; Yang et al. (2025) . We believe that trained, on-device sized agents are an exciting

Read the complete text