NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generation

Hossain Shaikh Saadi, Faria Alam, Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense · 2025

Read NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generation by Hossain Shaikh Saadi, Faria Alam, Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16787.txt.

Introduction

Hossain Shaikh Saadi, Faria Alam, Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, and Katharina von der Wense describe the winning entry to a shared task on writing Python code from Bangla instructions. Their system splits the job between two agents that pass work back and forth. One agent writes a first program from the Bangla prompt.

A second agent then runs the test cases, and only the failed cases move on for repair. Using the error traces, the current program, and the failed tests, it writes a fixed version. The loop earned first place in the BLP-2025 task with a top score on the main pass measure. And the team has released its code in public for others to inspect and reuse.

Catalog record

CREATORHossain Shaikh Saadi, Faria Alam, Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-16787

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generation Hossain Shaikh Saadi, Faria Alam, Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager, Katharina von der Wense Abstract This paper presents the winning system for the BLP-2025 Shared Task on Code Generation from Bangla Instructions, which consists of a multi-agent pipeline. First, a code-generation agent produces an initial solution from the input instruction. The candidate program is then executed against the provided unit tests (pytest-style, assert-based). Only the failing cases are forwarded to a debugger agent, which reruns the tests, extracts error traces, and, conditioning on the error messages, the current program, and the relevant test cases, generates a revised solution. Using this approach, our submission achieves first place in the shared task with a P ​ a ​ s ​ s ​ @ ​ 1 Pass@1 score of 95.4. We make our code publicly available. 1 1 1 https://github.com/shaikhsaadi999/blp25_code_genneration 1 Introduction In recent years, the quality of automatically generated code based on natural language instructions has increased rapidly, and a plethora of work exists in this domain Yang et al. (2025) ; Hui et al. (2024) ; Li et al. (2023) ; Huynh and Lin (2025) . However, most benchmarks and systems remain vastly English-centric Jiang et al. (2024) . This imbalance narrows access to program-synthesis tools for non-English speakers and limits our understanding of how linguistic factors – such as morphology, script variation, and code-mixing – impact the path from instructions to executable programs. Additionally, it makes it harder for developers with limited English proficiency to use these tools Wang et al. (2023) . Bangla – spoken by over 270 million people worldwide – is an example of an inadequately supported language in this area. Although NLP resources are expanding Raihan et al. (2025a) ; Raihan et al. (2025b) , there are still few high-quality evaluations and systems for Bangla instruction-to-code generation Raihan et al. (2025b) . This paper presents the winning system for the BLP-2025 Code Generation Shared Task Raihan et al. (2025c) . Our system adopts a targeted two-agent pipeline. First, a code-generation agent produces an initial Python solution from the Bangla instruction. We then execute the generated code against a set of pytest-style unit tests from the provided dataset and an external dataset Austin et al. (2021) to obtain concrete failure signals. Rather than re-prompting every data sample, only the failing samples together with the error traces, the current code, and the corresponding unit tests are passed to a debugger agent that proposes minimal, correctness-oriented edits. By localizing debugging to the right spots, we concentrate our inference effort and avoid needless code changes. We evaluate multiple proprietary APIs for this multi-agent approach. Empirically, this approach delivers strong results on the shared task data. Our contributions are two-fold: (i) a simple but effective agent architecture that couples generation with test-driven refinement; (ii) a selective feedback mechanism that exposes the debugger only to failing cases and distilled execution traces.On the official leaderboard, our system achieves the best performance among all participating teams (Pass@1: 95.4%). We complement the results with a small ablation study and analysis. An overview of our proposed system is provided in Figure 1 . Figure 1: Multi-agent Bangla \to Python code generation pipeline with selective debugging via unit test feedback. 2 Related Work Code generation with large language models (LLMs) is a well-established area of study Jiang et al. (2024) ; Dong et al. (2025) ; Qi et al. (2021) , with interest and capability surging in recent years Yang et al. (2025) ; Hui et al. (2024) ; Li et al. (2023) . Despite this momentum, there remains relatively little work on Bangla instruction-to-code generation Raihan et al. (2025b) . Test-driven development (TDD) is a well-established methodology in which developers write tests before implementing the functional code, ensuring that the resulting solution directly aligns with the initial problem statements Mathews and Nagappan (2024) . In TDD, test cases are the central artifact, and there is a growing body of research on automated test-case generation Takerngsaksiri et al. (2024) ; He et al. (2025b) ; Wang et al. (2025) ; He et al. (2025a) . For evaluating code generation systems, unit test-based evaluation has become the de facto standard Chen et al. (2021) . Rather than relying on text-only metrics used in machine translation (e.g., BLEU), running generated programs against hidden tests directly measures functional correctness Huynh and Lin (2025) . 3 Task Definition The goal of the shared task is to generate Python code from a given Bangla instruction. The instruction itself contains the given Bangla instruction, function name, and argument names. Furthermore, assert-based pytest-style unit tests Raihan et al. (2025c) are provided for each instruction. For each instance i , we are given a Bangla natural-language instruction x_{i}\in X which includes a function name and its argument names, and a pytest test suite T_{i} . No type signature is provided for the arguments of the given function; required behavior is determined solely by the instructions x_{i} and test suites T_{i} . A candidate program must define the function precisely as provided by the function name since pytest is designed with the same function names. For example, if a function’s name is min_cost , for this function, one given assert-based unit test will be like this, assertmin_cost([[1,2,3],[4,8,2],[1,5,3]],2,2)==8 . A program passes if all tests in T_{i} succeed, and the goal is to generate a program \hat{p}_{i} for an instruction x_{i} that passes all tests. For this task, the evaluation metric is Pass@1 Chen et al. (2021) . This is a standard metric for code generation tasks that measures the proportion of instructions for which a single generated code passes all the provided unit tests. So, in short, given one attempt, how often does the model produce an entirely correct solution. In the most common setting, for a given instruction, one solution per problem is generated, and then the fraction of the generated solutions that pass all the unit tests is reported as a Pass@1 score. 4 Data Development dataset Our development set consists of 400 Bangla instructions paired with function names, each accompanied by three unit tests. The organizers provide this dataset Raihan et al. (2025a) . Test dataset The test set contains 500 Bangla instructions with function names, each accompanied by a single unit test, so that we gain some understanding of the behavior of the input and output of the function we need to generate. Final submitted codes are tested on hidden unit tests for the sake of fair and robust evaluation. External dataset The task permits the use of external models and datasets. To augment the available tests for code generation, we search for a resource with matching function formats and existing unit tests. We identify Austin et al. (2021) , which directly covers unit tests for 480 relevant samples. In this external dataset, the number of unit tests per instance varies: some instances are provided with three tests, while others are provided with four. For each instruction in the provided data, we match its function name against those in Austin et al. (2021) , retrieve the corresponding 480 functions, and extract all associated unit tests.

Read the complete text