Development of an LLM-Based System for Automatic Code Generation from HEP Publications

Junichi Tanaka · 2026

Read Development of an LLM-Based System for Automatic Code Generation from HEP Publications by Junichi Tanaka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14696.txt.

Introduction

Saito, Kishimoto, and Tanaka built a trial system that reads high-energy physics papers, extracts their event-selection rules, and turns them into runnable code. Tested on a published ATLAS Higgs search in four-lepton events from 2015 and 2016 open data, it recovered many documented cuts and sometimes matched a hand-written baseline exactly. Randomness, invented details, and failed runs still limit it, so they present it as a supervised aid, not an autonomous analyst.

Catalog record

CREATORJunichi Tanaka
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23565
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14696

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Development of an LLM-Based System for Automatic Code Generation from HEP Publications Junichi Tanaka Abstract Ensuring the reproducibility of physics results is one of the crucial challenges in high-energy physics (HEP). In this study, we develop a proof-of-concept system that uses large language models (LLMs) to extract analysis procedures from HEP publications and generate executable analysis code for reproducing published results. Our method consists of two stages. In the first stage, open-weight LLMs extract event selection criteria, object definitions, and other relevant analysis information from a target paper and, when necessary, from its referenced publications, and then produce a structured selection list. In the second stage, the structured selection list is used to generate analysis code, which is then executed and validated iteratively. As a benchmark, we use the ATLAS H → Z ​ Z ∗ → 4 ​ ℓ H\rightarrow ZZ^{*}\rightarrow 4\ell analysis based on proton-proton collision data recorded in 2015 and 2016 and released as ATLAS Open Data. This benchmark allows direct comparison between the generated results and the published analysis, as well as comparison with a manually developed baseline implementation. We separately evaluate selection extraction and code generation in order to clarify the current capabilities and limitations of open-weight LLMs for HEP analysis reproduction. Our initial results show that recent open-weight models can recover many documented selection criteria from papers and references, and that in some runs they can generate event selections fully matching a baseline implementation at the event level. At the same time, stochasticity, hallucination, and execution failure remain significant challenges. These results suggest that LLMs are already promising as human-in-the-loop tools for reproducibility support, although they are not yet reliable as fully autonomous HEP analysis agents. In this paper, we report the design of the prototype system and its initial performance evaluation. [a]Masahiko Saito 1 Introduction Data analysis in high energy physics (HEP) has become increasingly complex, demanding substantial computational expertise and time to set up environments and write code. This situation raises the barrier to entry for students and newcomers. Large language models (LLMs) offer powerful support for coding and analysis frameworks, potentially allowing researchers to focus more on underlying physics problems. However, due to stochastic variation and hallucination, fully automated analyses remain difficult to trust without careful verification. Therefore, a human-in-the-loop framework is essential for applying LLMs to analysis support. In this study, we develop a proof-of-concept (PoC) LLM workflow pipeline that extracts analysis procedures from HEP publications and automatically generates executable analysis code ( fig. 1 ). The proposed method consists of two main stages implemented with open-weight LLMs. First, the system extracts selection criteria from the paper PDF and cited references to iteratively organize a structured selection list. Second, this list is used as input to iteratively generate, execute, and verify analysis code. Rather than directly generating code from the text, our approach introduces a human-readable intermediate representation, positioning the LLM as a verifiable collaborator rather than a black-box system. Furthermore, this system functions as a framework for assessing the reproducibility of HEP publications; successful reproduction suggests sufficient documentation, while failure may indicate missing or ambiguous descriptions. In the longer term, this tool could be used before publication to improve paper quality. Our main contributions are twofold. First, we implement a workflow that extracts structured selection criteria from papers and iteratively generates, executes, and validates analysis code. Second, using the ATLAS Open Data H\rightarrow ZZ^{*}\rightarrow 4\ell analysis as a benchmark, we separately evaluate paper understanding and code generation to quantitatively clarify the strengths and limitations of open-weight LLMs. The remainder of this paper details related work ( section 2 ), the benchmark design ( section 3 ), selection extraction ( section 4 ), code generation ( section 5 ), and conclusions with future directions ( section 6 , section 7 ). Figure 1 : Overview of the proposed two-stage workflow. Step 1 iteratively extracts and merges selection criteria from the target paper and its references. Step 2 utilizes these extracted criteria to sequentially generate, execute, and validate analysis code until successful reproduction is achieved. 2 Related work In recent years, large language models (LLMs) have been increasingly applied to HEP workflow automation. For instance, Esmail et al. [ 9 ] developed CoLLM, an end-to-end analysis environment using open-weight models. Gendreau-Distler et al. [ 10 ] proposed an agent-based framework integrated with Snakemake for ATLAS Open Data analysis, while Moreno et al. [ 14 ] demonstrated end-to-end execution of multiple analyses using Claude Code and literature-based retrieval. Unlike these studies, which focus on general end-to-end automation, our work specifically aims to extract analysis procedures directly from existing HEP publications to quantitatively evaluate reproducibility. Furthermore, we distinguish our approach by explicitly separating the workflow into verifiable document understanding and code generation stages using LangChain and LangGraph, while focusing primarily on the capabilities of open-weight models. 3 Benchmark Design and Evaluation Protocol As our benchmark, we selected the ATLAS H\to ZZ^{*}\to 4\ell analysis [ 2 ] . Because this paper delegates detailed event selection definitions to cited references, it is ideal for testing an LLM’s ability to trace and integrate multi-document information. The evaluation utilizes 2015–2016 DAOD_PHYSLITE ATLAS Open Data [ 5 , 6 , 4 , 3 ] . We manually reproduced the analysis to establish a baseline. While missing variables in the Open Data and omitted paper details prevented a perfect match with the published distribution ( fig. 2 ), the overall result was reasonably consistent. We use this manual implementation and a curated list of 27 explicitly identifiable selection criteria as our ground truth (Appendix A ). To clearly identify performance bottlenecks, we evaluate the two workflow stages separately. Step 1 (selection extraction) is evaluated against the 27 ground-truth cuts, using the number of correctly extracted cuts and contradictory hallucinations as metrics. Step 2 (code generation) evaluates only the cuts implementable via the Open Data Ntuple. The generated code is compared to the baseline implementation at the event level, categorizing results as Exactly Matched (identical event list), Not Matched (valid execution but differing events), or Execution Failed . (a) Published paper [ 1 ] (b) Manually reproduced distribution Figure 2 : Comparison of the four-lepton invariant mass distribution between (a) the published result [ 1 ] and (b) our manually reproduced baseline. 4 Step 1: Selection Extraction The goal of Step 1 is to extract event-selection criteria and object definitions from the target paper and its cited references. Because critical definitions are often delegated to citations, this multi-document extraction is essential for completeness. Rather than a simple numerical list, the extracted information is maintained as a structured representation containing comments and reference provenance (Appendix B ). This format facilitates downstream information integration and provides a verifiable, human-readable intermediate result. 4.1 Workflow Design The selection extraction system is an iterative workflow consisting of four components ( fig. 1 ). First, the Planner determines the next reference to read and formulates a specific reading objective based on the current selection list. This targeted approach prevents the LLM from extracting noisy or irrelevant information from supplementary references. Next, the Loader converts the PDF to Markdown using marker [ 13 ] , uses an LLM to isolate relevant body text, and maps in-text citations

Read the complete text