OmniNova:A General Multimodal Agent Framework
Du Pengfei · 2025
Read OmniNova:A General Multimodal Agent Framework by Du Pengfei (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20028.txt.
Introduction
Pengfei Du wrote this 2025 paper presenting OmniNova, a framework for teams of language-model agents that divide complex work among themselves. Single models with tools can automate tasks, but several agents together stumble over coordination, wasted resources, and garbled hand-offs. OmniNova organizes them into a hierarchy: coordinator, planner, supervisor, and specialist agents, each with a defined remit.
Tasks route dynamically to agents matched to their complexity, and different cognitive jobs draw on different sizes of model, so cheap thinking is not billed at premium rates. Across fifty tasks in research, data analysis, and web interaction, the framework beat a baseline on completion, 87 percent against 62, while using 41 percent fewer tokens. Human judges also scored its results higher. Du offers the design, plus an open-source implementation, as a template for automation systems.
Catalog record
| CREATOR | Du Pengfei |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20028
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniNova:A General Multimodal Agent Framework Du Pengfei Abstract The integration of Large Language Models (LLMs) with specialized tools presents new opportunities for intelligent automation systems. However, orchestrating multiple LLM-driven agents to tackle complex tasks remains challenging due to coordination difficulties, inefficient resource utilization, and inconsistent information flow. We present OmniNova, a modular multi-agent automation framework that combines language models with specialized tools such as web search, crawling, and code execution capabilities. OmniNova introduces three key innovations: (1) a hierarchical multi-agent architecture with distinct coordinator, planner, supervisor, and specialist agents; (2) a dynamic task routing mechanism that optimizes agent deployment based on task complexity; and (3) a multi-layered LLM integration system that allocates appropriate models to different cognitive requirements. Our evaluations across 50 complex tasks in research, data analysis, and web interaction domains demonstrate that OmniNova outperforms existing frameworks in task completion rate (87% vs. baseline 62%), efficiency (41% reduced token usage), and result quality (human evaluation score of 4.2/5 vs. baseline 3.1/5). We contribute both a theoretical framework for multi-agent system design and an open-source implementation that advances the state-of-the-art in LLM-based automation systems. I Introduction As Large Language Models (LLMs) have demonstrated increasing capabilities in natural language understanding and generation [ 1 , 2 , 3 ] , there has been growing interest in extending their functionality through integration with specialized tools [ 4 , 5 ] . These tool-augmented LLMs can perform actions beyond text generation, such as executing code [ 6 ] , retrieving information [ 7 ] , and interacting with external systems [ 8 ] . While impressive in isolation, employing a single LLM instance for complex tasks that require diverse capabilities presents significant limitations. For instance, research tasks often involve searching for information, analyzing data, and synthesizing findings—activities that benefit from specialized approaches. This has motivated the development of multi-agent systems where multiple LLM instances, each with specific roles and capabilities, collaborate to tackle complex problems [ 9 , 10 , 11 ] . Despite advances in this domain, existing multi-agent frameworks face several challenges: • Coordination Overhead : Managing interactions between multiple agents introduces significant complexity and potential for miscommunication [ 12 ] . • Resource Inefficiency : Using high-capability models for all tasks wastes computational resources [ 13 ] . • Planning Limitations : Many systems struggle with effective task decomposition and strategy formulation [ 14 ] . • Tool Integration : Seamlessly incorporating external tools remains challenging [ 5 ] . To address these limitations, we introduce OmniNova, a modular multi-agent automation framework that enables effective collaboration between LLM-based agents with diverse capabilities. OmniNova implements a hierarchical architecture where specialized agents are coordinated by supervisor agents that manage the overall workflow. This approach allows for efficient task decomposition, appropriate delegation, and effective synthesis of results. Our contributions include: 1. A hierarchical multi-agent architecture with distinct coordinator, planner, supervisor, and specialist agents, enabling effective task decomposition and delegation. 2. A dynamic task routing mechanism that optimizes agent deployment based on task complexity, reducing computational costs while maintaining performance. 3. A multi-layered LLM integration system that allocates models with appropriate capabilities to different cognitive requirements, balancing performance and efficiency. 4. A comprehensive evaluation demonstrating that OmniNova outperforms existing frameworks across multiple metrics and task types. 5. An open-source implementation that serves as both a practical tool and a foundation for future research on multi-agent systems. In the following sections, we provide a detailed overview of OmniNova’s architecture, implementation, and evaluation. We begin with a review of related work (Section II ), followed by a description of OmniNova’s system architecture (Section III ). We then detail our implementation (Section IV ) and evaluation methodology and results (Section V ). Finally, we discuss the implications and limitations of our work (Section VI ) and conclude with future directions (Section VII ). II Related Work II-A Large Language Models and Tool Use Large Language Models have evolved from basic text generation models to sophisticated systems capable of complex reasoning and problem-solving [ 1 , 15 , 16 ] . Recent work has focused on enhancing LLMs with the ability to use external tools, thereby extending their capabilities beyond their training data and enabling them to interact with the world [ 4 , 5 , 17 ] . Toolformer [ 4 ] pioneered this approach by fine-tuning language models to use external tools through API calls embedded in text. ToolLLM [ 5 ] extended this concept by developing a benchmark for tool-use and proposing improved training methodologies. ReAct [ 18 ] introduced a framework that interleaves reasoning and acting, enabling LLMs to formulate plans and execute actions in an integrated manner. Voyager [ 19 ] demonstrated an LLM-powered agent capable of autonomous exploration and skill acquisition in Minecraft through tool use. While these systems show promising capabilities, they typically rely on a single LLM instance, which limits their ability to handle complex tasks requiring diverse expertise and introduces efficiency challenges when using high-capability models for all tasks. II-B Multi-Agent Systems Multi-agent systems composed of LLM-based agents represent a growing research area aimed at addressing the limitations of single-agent approaches [ 11 , 10 , 9 ] . These systems distribute complex tasks across multiple specialized agents, enabling more effective problem-solving through collaboration. MetaGPT [ 11 ] implemented a multi-agent framework for software development, where different agents take on roles such as product manager, architect, and programmer. AutoGen [ 9 ] presented a framework for building applications with multiple conversational agents that can work together on tasks. ChatDev [ 10 ] demonstrated a virtual software development team composed of LLM agents that collaborate to build software applications. These systems demonstrate the potential of multi-agent approaches but often face challenges in effective coordination, efficient resource utilization, and seamless integration of external tools. II-C Agent Orchestration and Workflows Orchestrating multiple agents to work effectively on complex tasks requires sophisticated workflow management. LangGraph [ 20 ] and CrewAI [ 21 ] have emerged as frameworks for defining and managing agent workflows, enabling the creation of complex interaction patterns and decision processes. CAMEL [ 22 ] explored communication and collaboration between artificial intelligence agents by implementing a framework for role-playing scenarios. CHATGPT-TEAM-GEN [ 23 ] demonstrated how multiple instances of ChatGPT could collaborate as a team to solve complex tasks. Task-specific frameworks like AgentVerse [ 24 ] for simulation environments and Ghost [ 25 ] for distributed task execution have further advanced the field. Our work builds upon these foundations while addressing key limitations in coordination efficiency, resource utilization, and adaptability to varying task complexities. II-D Hierarchical Planning and Execution The concept of hierarchical planning and execution, where complex tasks are decomposed into subtasks that are assigned to specialized agents, has been explored in various contexts [ 26 , 27 , 28 ] . CLAIRIFY [ 29 ] demonstrated how language model agents can decompose complex information-seeking tasks into hierarchical steps. ReAct-Plan-Solve [ 30 ] proposed a framework for task planning and execution that combines planning, action, and dialogue. ViperGPT [ 31 ] implemented a system that generates and executes programs for visual reasoning tasks. OmniNova extends these approaches with a more comprehensive hierarchical system that incorporates dynamic routing, multi-layered model allocation,