Automated File-Level Logging Generation for Machine Learning Applications using LLMs: A Case Study using GPT-4o Mini

Mayra Sofia Ruiz Rodriguez, SayedHassan Khatoonabadi, Emad Shihab · 2025

Read Automated File-Level Logging Generation for Machine Learning Applications using LLMs: A Case Study using GPT-4o Mini by Mayra Sofia Ruiz Rodriguez, SayedHassan Khatoonabadi, Emad Shihab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04820.txt.

Introduction

Mayra Ruiz Rodriguez, SayedHassan Khatoonabadi, and Emad Shihab automate logging in twenty twenty-five. Log statements help developers watch system behavior and debug faults. Language models can write code, so they may also write logs. Prior studies judged logs inside single functions, leaving whole-file logging unexplored.

Machine learning projects are their test case, where thorough logs aid reliability. The team gathered one hundred seventy-one ML repositories with over four thousand Python files. They removed the original logs from those files. GPT-4o mini was then prompted to regenerate logs at file level. Both the placement and quality of the new logs were evaluated. Readers learn how far a small model goes on a tedious, real coding chore.

Catalog record

CREATORMayra Sofia Ruiz Rodriguez, SayedHassan Khatoonabadi, Emad Shihab
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04820

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automated File-Level Logging Generation for Machine Learning Applications using LLMs: A Case Study using GPT-4o Mini Mayra Sofia Ruiz Rodriguez, SayedHassan Khatoonabadi, Emad Shihab Abstract Logging is essential in software development, helping developers monitor system behavior and aiding in debugging applications. Given the ability of large language models (LLMs) to generate natural language and code, researchers are exploring their potential to generate log statements. However, prior work focuses on evaluating logs introduced in code functions, leaving file-level log generation underexplored—especially in machine learning (ML) applications, where comprehensive logging can enhance reliability. In this study, we evaluate the capacity of GPT-4o mini as a case study to generate log statements for ML projects at file level. We gathered a set of 171 ML repositories containing 4,073 Python files with at least one log statement. We identified and removed the original logs from the files, prompted the LLM to generate logs for them, and evaluated both the position of the logs and log level, variables, and text quality of the generated logs compared to human-written logs. In addition, we manually analyzed a representative sample of generated logs to identify common patterns and challenges. We find that the LLM introduces logs in the same place as humans in 63.91% of cases, but at the cost of a high overlogging rate of 82.66%. Furthermore, our manual analysis reveals challenges for file-level logging, which shows overlogging at the beginning or end of a function, difficulty logging within large code blocks, and misalignment with project-specific logging conventions. While the LLM shows promise for generating logs for complete files, these limitations remain to be addressed for practical implementation. I Introduction Logging plays a critical role in modern software development, enabling monitoring in production environments [ 1 ] , failure diagnosis [ 2 , 3 ] , and facilitating performance analysis [ 4 ] . For effective logging, developers need to decide what-to-log and where-to-log. What-to-log refers to choosing the appropriate log level (e.g., warning, error), text, and variables, while where-to-log concerns the location in the code where the log should be introduced. With the advent of large language models (LLMs), researchers have seen promising results in various software development tasks, including code generation [ 5 ] , testing [ 6 ] , and resolving bugs [ 7 ] . Recently, researchers have worked on the non-trivial task of generating complete log statements using LLMs. Mastropaolo et al. [ 8 ] introduced LANCE and later LEONID [ 9 ] , both models trained on millions of methods to generate log statements at the method level. However, the fine-tuning process for these models requires significant computational resources. To reduce training demands, Xu et al. [ 10 ] proposed a framework that uses an in-context learning prompt paradigm with a few carefully selected examples to guide the model to generate logs. Building on this direction, Li et al. [ 11 ] evaluated different LLMs’ performance on log generation. Their study involves removing log statements from functions and using placeholders where logs should be introduced before prompting the LLM. Prior studies have primarily focused on inserting a single log within a method, without providing the LLM with the full context of the file and without considering the possibility of inserting multiple logs. Furthermore, compared to traditional software, machine learning (ML) applications face distinct characteristics: there is an inherent dependency on data, and their non-deterministic behavior makes it difficult to debug. ML applications show lower log statements than traditional software, averaging one log statement per 1150 lines of code in ML applications using Python [ 12 ] . As a result, using LLMs for this purpose is not systematically explored. In this study, we aim to investigate how GPT-4o mini performs in the task of logging complete ML files, without specifying any placeholder in the code. To this end, we first curate a list of 371 ML projects on GitHub. We extract all Python files from each project and determine which Python files have at least one log, reducing our dataset to 171 projects. We collect a dataset consisting of 4,073 Python files, then we remove the logs in the files and prompt GPT-4o mini to generate logs for these files. Based on this dataset, we aim to answer the following three research questions: RQ1: To what extent does GPT-4o mini log in the same position as human written logs? We aim to understand human and LLM logging patterns because, while logging is essential for debugging, LLMs’ tendency toward verbose outputs [ 13 ] might result in excessive logging. We compared the position of logs for both human-written and GPT-4o mini generated logs. We find that the LLM prioritizes comprehensive logging coverage, as shown by its low underlogging rate (4.75%) and moderately high coverage (63.91%). However, that results in generating a large number of logs: 5.15 times more than those written by humans. Even in cases where both humans and the LLM logged, the LLM tends to include more logs overall, with a rate of 68.03%. RQ2: How does GPT-4o mini perform at logging ML applications compared to human developers? We aim to understand the quality of LLM-generated logs compared to human-written logs. To this end, we analyze the generated logs in terms of ingredients: level, variables, and text; and compare them to the corresponding human-generated log. We find that GPT-4o mini shows moderate quality but significant limitations when generating logs compared to human developers. While it matches the log levels with 59.19% exact match and 84.34% average ordinal distance, it identifies only 40.58% of variables that human logged. Its log texts are similar in meaning to human logs (ROUGE-L: 0.316), but uses different vocabulary and phrasing (BLEU-4: 0.050), and require substantial editing to match human logs (Levenshtein: 0.735). RQ3: What are the challenges of using GPT-4o mini for automated log generation? We aim to identify GPT-4o mini’s limitations in file-level logging. For this purpose, we categorize all developer-LLM pairs of logs into four categories: overlogging, underlogging, different levels, and different variables. We get a stratified sample of 384 pairs of logs, and manually analyze the surrounding code in which they were introduced. We find that GPT4-o mini’s main challenge is overlogging (85.8%), with most logs introduced at the start or end of a function. Moreover, the LLM underlogs (4.7%), particularly in large code blocks. In cases with different log level (5.3%), the LLM does not align with project-specific logging conventions. Finally, in case of different variables captured between humans and the LLM (4.3%), the LLM sometimes misses important variables that are present in the GT, despite the larger available context. Based on our findings, we discuss key considerations for using GPT-4o mini in logging complete ML files. While LLMs can generate useful log statements, they often overlog, potentially cluttering code and reducing its effectiveness for debugging. Developers should critically review LLM-generated logs to ensure relevance and avoid excessive logging. Additionally, repository-specific logging configurations, such as custom log levels, are generally overlooked but crucial for log generation in real-world projects. Our results suggest that future tools might benefit from this information to generate higher-quality logs that aligns better with previous developers’ preferences. Our Contributions. In summary, we make the following contributions in

Read the complete text