Multilingual Multimodal Software Developer for Code Generation

Linzheng Chai, Jian Yang, Shukai Liu, Wei Zhang, Liran Wang, Ke Jin, Tao Sun, Congnan Liu, Chenchen Zhang, Hualei Zhu, Jiaheng Liu, Xianjie Wu, Ge Zhang, Tianyu Liu, Zhoujun Li · 2025

Read Multilingual Multimodal Software Developer for Code Generation by Linzheng Chai, Jian Yang, Shukai Liu, Wei Zhang, Liran Wang, Ke Jin, Tao Sun, Congnan Liu, Chenchen Zhang, Hualei Zhu, Jiaheng Liu, Xianjie Wu, Ge Zhang, Tianyu Liu, Zhoujun Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08719.txt.

Introduction

A coder model that reads diagrams as well as text is built by Linzheng Chai and co-authors in a 2025 arXiv paper. Real software work leans on UML charts and flowcharts, which text-only models miss. Their MM-Coder takes those visual plans plus words and writes code to match. A new teach set pairs charts with code tasks for training. A new bench, MMEval, grades chart-led code work.

Even strong models stumble on fine chart detail and rule keeping, the bench shows. The gap to human-like chart reading stays wide. The sets are shared for the field.

Catalog record

CREATORLinzheng Chai, Jian Yang, Shukai Liu, Wei Zhang, Liran Wang, Ke Jin, Tao Sun, Congnan Liu, Chenchen Zhang, Hualei Zhu, Jiaheng Liu, Xianjie Wu, Ge Zhang, Tianyu Liu, Zhoujun Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08719

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multilingual Multimodal Software Developer for Code Generation Linzheng Chai, Jian Yang, Shukai Liu, Wei Zhang, Liran Wang, Ke Jin, Tao Sun, Congnan Liu, Chenchen Zhang, Hualei Zhu, Jiaheng Liu, Xianjie Wu, Ge Zhang, Tianyu Liu, Zhoujun Li Abstract The rapid advancement of Large Language Models (LLMs) has significantly improved code generation, yet most models remain text-only, neglecting crucial visual aids like diagrams and flowcharts used in real-world software development. To bridge this gap, we introduce M 2 -Coder , a Multilingual Multimodal software developer. M 2 -Coder integrates visual design inputs—Unified Modeling Language (UML) diagrams and flowcharts (termed Visual Workflow)—with textual instructions to enhance code generation accuracy and architectural alignment. To enable this, we developed M 2 c-Instruct , a diverse multimodal instruction-tuning dataset including visual-workflow-based code generation, allowing M 2 -Coder to synthesize textual and graphical information like human developers, distinct from prior work on narrow tasks. Furthermore, we introduce M 2 Eval , a new benchmark for evaluating multimodal code generation, addressing existing text-only limitations. Our evaluations using M 2 Eval highlight significant remaining challenges for models in precise visual information capture, instruction following, and advanced programming knowledge. Our work aims to revolutionize industrial programming by enabling LLMs to interpret and implement complex specifications conveyed through both text and visual designs. 1 Introduction The emergence of large language models (LLMs) and Large Multimodal Models(LMMs), such as Claude3 ( claude3 ) and GPT4o/GPT4.5 ( gpt4 ; gpt45 ) , has shown exceptional performance in various code-related tasks, particularly in code generation. Meanwhile, open-source code LLMs like StarCoder ( starcoder ) , DeepSeekCoder ( deepseek_coder ) also deliver competitive results on foundational text-only code generation benchmarks, such as LiveCodeBench ( livecodebench ) , Aider ( aider ) , SWE-Bench ( swe_bench ) , and BigCodeBench ( bigcodebench ) . However, these code LLMs involve processing exclusively textual inputs. This contrasts sharply with industrial programming, where translating complex requirements into functional code often relies heavily on visual aids like diagrams, UI mockups, and schematics to improve clarity and collaboration. There has been some recent work on visual input for code LMMs, such as Design2Code ( design2code ) , Web2Code ( web2code ) , MatPlotBench ( yang2024matplotagent ) , and ChartCoder ( zhao2025chartcoder ) . These methods focus on restoring the source code from the image input, including graph-to-code generation. Most previous works restore the functional code that can render the corresponding image, so the code generation task is limited only to the narrow field of code-based drawing (creating visuals using programming languages). In the industrial scenario, the software architect often first provides a visual representation of system architecture/design patterns/process workflows in Figure 1 , and then the software developer implements the actual code. Unlike human developers, who seamlessly integrate textual and graphical information to understand intricate specifications, current code LLMs operate solely on text-based inputs, limiting their effectiveness in scenarios where visual context is critical. To bridge the gap between current code generation models and the nuanced understanding of human software developers, we propose the M ultilingual M ultimodal software developer for Code generation ( M 2 -Coder ). This model is designed to effectively incorporate visual software designing information directly into the code generation pipeline. We utilize UML (Unified Modeling Language) diagrams and flowcharts as “Visual Workflow”, as these provide indispensable visual representations of system architecture, processes, and logic before coding commences. As depicted in Figure 1 , UML diagrams facilitate complex system design and documentation, while flowcharts clarify algorithmic logic, thus enabling M 2 -Coder to grasp higher-level design intent. Specifically, M 2 -Coder undergoes a two-stage training: initial pre-training on diverse large-scale multimodal code-related instruction samples, followed by fine-tuning on a high-quality instruction corpus emphasizing visual-workflow-based code generation. Finally, to accurately evaluate the performance of code LMMs like M 2 -Coder , we introduce a new benchmark, M 2 Eval . The contributions and insights are summarized as: • We introduce M 2 c-Instruct , a large-scale multilingual multimodal code generation dataset comprising over 13.1M samples across 50 programming languages. Building upon this, we propose M 2 -Coder , a novel multimodal code generation model trained on M 2 c-Instruct that leverages visual design inputs, such as UML diagrams and flowcharts, to improve code generation accuracy and alignment with architectural intent. • We present M 2 Eval , a new benchmark for evaluating multilingual multimodal code generation. M 2 Eval significantly expands the scope of evaluation in this field by incorporating a broader range of programming languages and diverse task types. • Our systematic evaluation of M 2 -Coder and popular LMMs on M 2 Eval demonstrates our 7B M 2 -Coder ’s competitiveness with larger 70B+ LMMs. Crucially, our analysis uncovers persistent LMM limitations in precise visual understanding, instruction following, and applying higher-order programming concepts, offering valuable directions for the field. Figure 1: Overview of M 2 -Coder , M 2 c-Instruct and M 2 Eval . We constructed M 2 c-Instruct , an instruction tuning dataset with over 13.1 million instances, to enhance the multilingual multimodal programming capabilities of M 2 -Coder . For evaluating M 2 -Coder , we curate M 2 Eval , a multimodal benchmark for code evaluation across 10 programming languages. 2 M 2 Eval Benchmark 2.1 Task Definition Given the k -th language L_{k} from the set {L_{1},...,L_{K}} of K programming languages(PLs), we input the instruction I^{L_{k}} and diagram D into the LMM \mathcal{M} to generate a code-related response c^{L_{k}} , sampled from the distribution P(c^{L_{k}}\mid I^{L_{k}},D;\mathcal{M}) . We then extract the code from c^{L_{k}} and execute it with the corresponding test cases u^{L_{k}} to obtain the final output. The output is expected to match the results specified by the test cases. The process can be described as: \displaystyle r^{L_{k}}=\mathbb{I}(P(c^{L_{k}}|I^{L_{k}},D;\mathcal{M});u^{L_{k}}) (1) where \mathbb{I}(\cdot) is the indicator function by executing the generated code with the given test cases u^{L_{k}} . when the generated code c^{L_{k}} passes all test cases, the evaluation result r=1 , else r=0 . 2.2 Data Curation Process Figure 2: The curation process for M 2 Eval : (1) designing Python prototype problems grounded in core programming concepts; (2) transforming these into multimodal problems by incorporating diagrams and refining prompts; and (3) translating the problems into multiple programming languages. As illustrated in Figure 2 , we follow a three-step process to build high-quality M 2 Eval : (1) design prototype problems in Python based on common programming concepts; (2) convert them into multimodal problems by adding diagrams and removing redundant text; and (3) translate them into multiple programming languages. A general overview is presented here, with further details in Appendix C . Prototype Problem Design. We design prototype Python problems based on scenarios and varying programming knowledge levels. An LLM generates initial prompts, solutions, and test cases. These are then manually refined through multiple iterations with the LLM to ensure accuracy, alignment with design goals, and comprehensive test coverage. Problem Design. Next, for each prototype problem, we generate diagrams ( PlantUML/Mermaid ) from the solution code using an LLM, then manually refine these diagrams for structural and semantic accuracy. We revise the problem prompts by removing information redundant with the diagrams, making the prompt alone insufficient for a correct

Read the complete text