Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We? Journal: TOSEM CCS: Software and its engineering Software maintenance tools

Kazuki Kusama, Honglin Shu, Masanari Kondo, Yasutaka Kamei · 2026

Read Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We? Journal: TOSEM CCS: Software and its engineering Software maintenance tools by Kazuki Kusama, Honglin Shu, Masanari Kondo, Yasutaka Kamei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25374.txt.

Introduction

Kazuki Kusama and colleagues test how well language models write log statements, the notes programs leave for debugging, across languages. Most earlier tools were judged in one language only, leaving multilingual performance unknown. The team builds a benchmark of 150,000 examples across five languages and compares three specialist tools with five language models. A specialist system called UniLog performs best overall and stays strong across languages. Difficulty varies by language, with Python hardest and JavaScript easiest, a gap tied to log habits rather than model size or data volume.

Catalog record

CREATORKazuki Kusama, Honglin Shu, Masanari Kondo, Yasutaka Kamei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112638
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25374

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Language Models for Log Statement Generation in Multilingual Scenarios: How Far Are We? Journal: TOSEM CCS: Software and its engineering Software maintenance tools Kazuki Kusama, Honglin Shu, Masanari Kondo, Yasutaka Kamei Abstract . Log statements capture critical information for software maintenance activities such as testing, debugging, and failure analysis. Because of this importance, developers must carefully design log statements, which requires significant effort. To support developers, various end-to-end automated log statement generation approaches have been proposed, whereas these approaches have mainly been evaluated within a single programming language environment and their effectiveness in multilingual environments remains underexplored. In this paper, we therefore comparatively evaluate three state-of-the-art log statement generation approaches and five large language models (LLMs) across multiple programming languages. For this purpose, we constructed a multilingual benchmark comprising 150,000 instances across five programming languages. Our empirical results demonstrate that UniLog, a state-of-the-art approach, achieves the best overall performance, maintaining high effectiveness even in multilingual environments. We also observe substantial variance in the difficulty of log generation across languages: Python presents a greater challenge, whereas JavaScript yields comparatively better performance. Detailed analysis reveals that these disparities stem from variations in log insertion distributions and language-specific logging idioms. Our findings indicate that simply scaling model size or the volume of training data is insufficient for multilingual log generation; rather, designing approaches tailored to the specific characteristics of target languages is crucial. These findings suggest that future automated logging techniques should explicitly account for language-specific logging characteristics to achieve robust performance in multilingual software development environments. 1. Introduction Integrating log statements into code is a fundamental practice for capturing the dynamic internal behavior of software systems. These log statements serve as an indispensable resource, driving critical maintenance activities such as testing, debugging, and failure analysis ( 7 ; 19 ) . To ensure that these log statements provide actionable insights while avoiding the system overhead of excessive logging and the information loss of insufficient logging, developers must carefully orchestrate their design by pinpointing optimal insertion locations, assigning appropriate severity levels, and formulating context-rich messages. Because this design process inherently relies on the manual judgment of developers, it is notoriously challenging and imposes a substantial burden on developer time and effort ( 61 ; 15 ; 64 ; 26 ) . To alleviate the burden of manual log statement design, prior research has actively explored automated log statement generation ( 18 ) . Previously, this process was typically decomposed into discrete subtasks, such as identifying insertion locations, determining appropriate severity levels, and generating message content, and developed specialized automation techniques for each ( 32 ; 35 ) . Advances in language models have led researchers to propose end-to-end approaches that seamlessly integrate these subtasks to holistically automate log generation ( 37 ; 56 ; 57 ) . For instance, UniLog ( 57 ) is an end-to-end approach for log statement generation that combines retrieval-based few-shot prompting with a warmup strategy to achieve strong performance. Despite significant advances in automated log statement generation with language models, current evaluation approaches still suffer from three important limitations: (I) The performance of log statement generation approaches in multilingual environments remains underexplored. Existing end-to-end log statement generation studies have been evaluated primarily on a single programming language, especially Java, and it remains unclear whether these approaches are equally effective in multilingual environments ( 37 ; 56 ; 57 ) . Modern software development commonly involves the use of multiple programming languages, such as Python and JavaScript ( 28 ) . This makes it necessary to move beyond single-language settings and evaluate the performance of log statement generation approaches across multiple programming languages. (II) The impact of training strategies on multilingual log statement generation remains unclear. Existing log statement generation approaches adopt different training and adaptation strategies, such as fine-tuning pre-trained language models, retrieval-based prompting, and lightweight warmup. In multilingual environments, it remains unclear whether models should be adapted separately for each programming language or trained jointly across multiple languages. Language-specific adaptation may better capture local logging practices, vocabulary, and coding styles, whereas multilingual training may help models learn common patterns shared across languages. Existing studies have not sufficiently investigated how such training strategies affect performance in multilingual log statement generation. Without clarifying this point, determining how training data and model adaptation should be designed for practical multilingual log statement generation remains difficult. (III) It remains unclear why automated log statement generation exhibits different performance across programming languages. Even if multilingual evaluation reveals performance differences across programming languages, such results alone do not explain why prediction is easier in some languages and more difficult in others. Log statement generation involves multiple decisions, including where to insert logs, which log levels to assign, and what message content to generate. These decisions may be influenced by language-specific factors, such as control structures, library usage, coding styles, and logging idioms. However, existing studies have not sufficiently analyzed how such language-specific logging characteristics are related to performance differences across programming languages. Without clarifying these factors, it remains difficult to explain cross-language performance gaps and to derive design guidelines for log statement generation approaches suitable for multilingual environments. To address these three limitations, this study empirically investigates end-to-end log statement generation in multilingual environments. Specifically, we construct a comprehensive multilingual benchmark from public GitHub repositories across five programming languages: Java, Python, JavaScript, TypeScript, and C#. For each language, the benchmark contains 24,000 training instances, 3,000 validation instances, and 3,000 test instances, resulting in a total of 150,000 instances. Using this benchmark, we comparatively evaluate the performance of three representative existing approaches and five large language models (LLMs) under unified experimental conditions, including the same target languages, data construction criteria, and evaluation metrics. We also analyze the effects of different training strategies and examine language-specific logging characteristics, including log position, log level, and log message, thereby clarifying cross-language differences and the factors underlying them. We define the following three research questions (RQs): (RQ1) What is the performance of log statement generation approaches in multilingual environments? Approach: We evaluate three representative existing approaches, LANCE ( 37 ) , FastLog ( 56 ) , and UniLog ( 57 ) . Since general-purpose LLMs have shown promising performance in log statement generation, we also evaluate five LLMs in multilingual environments: Llama3, Qwen2.5-Coder, Mistral, DeepSeek-V3, and GPT-4.1 mini. Results: UniLog achieved the highest performance among the existing approaches in multilingual environments and perfectly predicted log statements with an average of 20.35% across all languages. Among the evaluated LLMs, DeepSeek-V3 achieved the highest performance and perfectly predicted log statements with an average of 10.96% across all languages. In addition, JavaScript showed the highest overall performance, whereas Python showed the lowest overall performance, which indicates that prediction difficulty differs across languages. (RQ2) What is the impact of training strategies on log statement generation in multilingual environments? Approach: We evaluate multiple training strategies, including monolingual and multilingual training, warmup-based adaptation, and training strategies used in representative end-to-end approaches. Results: Single-language training showed overall better performance than mixed-language training. UniLog also showed strong performance, although it requires less training data. (RQ3) How do language-specific characteristics affect automated log statement generation in multilingual environments? Approach: We examine insertion locations, log levels, and message

Read the complete text