Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback
Xin Wang, Feng Yang, Xiaoqian Jiao, Yang Zhang, Zhenhao Li, Zishuo Ding · 2026
Read Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback by Xin Wang, Feng Yang, Xiaoqian Jiao, Yang Zhang, Zhenhao Li, Zishuo Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29122.txt.
Introduction
Wang and co-authors present ReLog, a system that writes logging code with runtime feedback. Good logs help developers and language models debug software. Older tools write logs in one pass from static code alone. They are judged by likeness to human-written logs. ReLog instead generates, runs, and refines logs in a loop. The team judges the logs by debugging tasks on the Defects4J benchmark. ReLog repaired 97 defects in the direct setting. Its F1 score beat all baselines tested.
Catalog record
| CREATOR | Xin Wang, Feng Yang, Xiaoqian Jiao, Yang Zhang, Zhenhao Li, Zishuo Ding |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56625 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29122
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback Xin Wang, Feng Yang, Xiaoqian Jiao, Yang Zhang, Zhenhao Li, Zishuo Ding Abstract . Logging statements are important for software debugging and maintenance. However, existing approaches to automatic logging statement generation primarily rely on static code analysis, generating statements in a single pass without considering actual program execution. Moreover, they typically evaluate generated statements by comparing them against developer-written ones, implicitly assuming that the original statements constitute an adequate gold standard. This assumption is increasingly restrictive in the LLM era: logging statements are consumed not only by human developers but also by LLMs for downstream tasks. As a result, generating logging statements solely for human developers, and evaluating them solely by similarity to developer-written statements, does not necessarily reflect their practical usefulness. To overcome these limitations, we introduce ReLog , an iterative logging generation framework guided by continuous runtime feedback. ReLog leverages LLMs to generate, execute, evaluate, and iteratively refine logging statements based on runtime feedback, so that the runtime logs provide sufficient information for downstream tasks. Instead of measuring similarity to developer-written statements, we evaluate ReLog through downstream tasks (i.e., defect localization and repair). We construct a new benchmark derived from Defects4J under two settings, namely direct and indirect debugging. The results show that ReLog consistently outperforms all baselines across both settings. In particular, it achieves the highest bug-detection F1 score of 0.520 and successfully repairs 97 defects in the direct setting, while also obtaining the best F1 score of 0.408 in the indirect setting where source code is unavailable. Additional experiments across multiple LLMs demonstrate the generality of the framework, and ablation studies confirm the importance of iterative refinement and compilation repair. Overall, our work reframes logging statement generation as a runtime-guided process that accounts for LLM-based downstream use, and advocates evaluating generated logging statements by their downstream utility rather than textual similarity. 1. Introduction Software developers insert logging statements into source code to expose useful information about program execution. During execution, these statements produce runtime logs that capture detailed system behaviors and states ( 30 ; 56 ) , thereby supporting a wide range of software engineering tasks, such as debugging, performance analysis, and system monitoring ( 52 ; 1 ; 41 ; 4 ; 25 ; 29 ; 19 ; 36 ) . As illustrated in Figure 1 , a typical logging statement consists of a severity level (e.g., info, debug), a static text template (e.g., “{} execution time: ms.”), and dynamic variables (e.g., method name, execution time), which together record the runtime information about a program event for subsequent analysis. Figure 1. The iterative refinement process of a logging statement in Apache HBase. Developers initially recorded execution time in nanoseconds but later revised the code to millisecond precision and a lower severity level based on actual runtime observations. Despite their importance, writing effective logging statements remains challenging for developers ( 5 ; 29 ; 19 ) . To alleviate this challenge, researchers have conducted a variety of studies aimed at improving logging quality and better supporting developers. Some investigate developers’ logging preferences and practices ( 5 ; 17 ; 6 ; 34 ; 33 ; 37 ; 14 ; 13 ; 32 ; 55 ; 47 ) , while others propose techniques for automatically suggesting or generating logging statements ( 57 ; 33 ; 35 ; 20 ; 11 ; 12 ; 31 ; 56 ) . These studies largely take a developer-centric perspective, focusing on what kinds of logging statements developers prefer and how to generate statements that resemble human-written ones. However, this perspective is becoming increasingly limited in the LLM era, where logging statements are consumed not only by human developers but also by LLMs for downstream tasks such as defect localization, failure analysis, and program repair ( 42 ; 51 ; 16 ; 26 ; 22 ) . Meanwhile, prior work ( 31 ; 56 ; 45 ) typically evaluates generated logging statements by comparing them against developer-written ones, implicitly assuming that the original statements constitute an adequate gold standard. This assumption may also be questionable in practice, because developer-written logging statements may themselves be suboptimal ( 5 ; 34 ; 13 ; 32 ; 47 ) , and similarity to such statements does not necessarily reflect their practical utility for log-based downstream tasks. Therefore, we argue that automatic logging statement generation should move beyond merely imitating developer-written statements and instead aim to produce logging statements that are actually useful for downstream tasks, while also accounting for the needs of LLMs as emerging consumers of runtime logs. In addition, existing approaches to automatic logging statement generation typically formulate the task as a static, single-pass generation problem, where the input is source code or other contextual information, such as abstract syntax trees, and the output is either a logging statement or a code snippet containing one. However, this formulation is inherently limited. In practice, logging statements are often refined iteratively based on the runtime logs they produce (cf. Section 2 ): developers inspect the generated runtime logs, identify deficiencies in the logging statement, and then revise its content, granularity, or severity level accordingly. For example, as illustrated in Figure 1 , in issue HBASE-24367 ( 18 ) , developers initially introduced a logging statement to record the average execution time of background chores in nanoseconds. However, the resulting runtime logs revealed that such fine-grained precision was neither intuitive for operators nor representative of the underlying performance characteristics. Consequently, the developers revised the logging statement to report execution time in milliseconds and adjusted the severity level to a more appropriate one. This refinement process highlights that effective logging often depends on runtime behaviors and execution states that cannot be fully inferred from static code context alone. As a result, static approaches may generate syntactically correct logging statements while still failing to capture the diagnostic information needed in practice. To address the above limitations, we propose ReLog , a runtime feedback-driven framework for iterative logging statement generation. Instead of treating logging statement generation as a static single-pass prediction task, ReLog formulates it as an execution-aware refinement process. Given a target program, ReLog first executes the original code to capture its execution outcome, and then generates initial logging statements based on both the code context and the observed runtime behavior. After inserting the generated statements, ReLog compiles and executes the instrumented program, using a repair loop to resolve any compilation errors introduced by the added logging statements. It then invokes an LLM-based module to assess whether the resulting runtime logs provide sufficient observability for downstream tasks. If the diagnostic evidence is still insufficient, ReLog employs a refiner module to update the logging statements based on the feedback, and repeats the process iteratively. In this way, ReLog goes beyond generating logging statements that merely resemble developer-written ones, and instead aims to produce logging statements whose resulting runtime logs are useful for LLM-based downstream tasks. To evaluate the practical utility of generated logging statements, we construct two new downstream debugging datasets from Defects4J ( 24 ) . These datasets are designed to assess whether generated logging statements provide sufficient diagnostic information for downstream debugging under different