SURGE : On the Potential of Large Language Models as General-Purpose Surrogate Code Executors

Bohan Lyu, Siqiao Huang, Zichen Liang · 2025

Read SURGE : On the Potential of Large Language Models as General-Purpose Surrogate Code Executors by Bohan Lyu, Siqiao Huang, Zichen Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11167.txt.

Description

Bohan Lyu, Siqiao Huang, and Zichen Liang ask if large language models can stand in for a code runner and predict how a run ends. Stand-in models save time in data work, and the new models shine at reading and writing code. Their 2025 bench, SURGE, holds 1,160 tasks in eight groups. Groups span code in many languages, hard contest problems, whole-store analysis, and costly science sums.

Retrieval and proof tasks join bug hunts and runs bound to one tool set. The team grades 21 models, open and commercial, on growth laws, data thrift, and prediction accuracy. Readings show where the models can fake a run well and where they slip. The bench and its grading frame sit open for all. Fast fake runs could cut the cost of code tests, if accuracy holds.

Catalog record

CREATORBohan Lyu, Siqiao Huang, Zichen Liang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE128 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11167

Browse the full library · About the Tower of Babel