MAFA: A Multi-Agent Framework for Annotation
Mahmood Hegazy, Aaron Rodrigues, Azzam Naeem · 2025
Read MAFA: A Multi-Agent Framework for Annotation by Mahmood Hegazy, Aaron Rodrigues, Azzam Naeem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13668.txt.
Introduction
Mahmood Hegazy, Aaron Rodrigues, and Azzam Naeem match bank customer questions to the right answers. Their paper, from twenty twenty-five, is called MAFA. Bank apps must link what a user types to the best item in a list of common questions. One model alone can miss the nuance in varied phrasing.
Their frame uses several agents, each with a different approach. Each agent reasons through set steps and gets its own worked examples. A judge agent then ranks the candidates and picks the best. On a real bank data set and two public sets, scores rose over single-agent runs. Top-one accuracy gained fourteen percent.
Catalog record
| CREATOR | Mahmood Hegazy, Aaron Rodrigues, Azzam Naeem |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13668
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAFA: A Multi-Agent Framework for Annotation Mahmood Hegazy, Aaron Rodrigues, Azzam Naeem Abstract Modern consumer banking applications require accurate and efficient retrieval of information in response to user queries. Mapping user utterances to the most relevant Frequently Asked Questions (FAQs) is a crucial component of these systems. Traditional approaches often rely on a single model or technique, which may not capture the nuances of diverse user inquiries. In this paper, we introduce a multi-agent framework for FAQ annotation that combines multiple specialized agents with different approaches and a judge agent that reranks candidates to produce optimal results. Our agents utilize a structured reasoning approach inspired by Attentive Reasoning Queries (ARQs), which guides them through systematic reasoning steps using targeted, task-specific JSON queries. Our framework features a few-shot example strategy, where each agent receives different few-shots, enhancing ensemble diversity and coverage of the query space. We evaluate our framework on a real-world major bank dataset as well as public benchmark datasets (LCQMC and FiQA), demonstrating significant improvements over single-agent approaches across multiple metrics, including a 14% increase in Top-1 accuracy, an 18% increase in Top-5 accuracy, and a 12% improvement in Mean Reciprocal Rank on our dataset, and similar gains on public benchmarks when compared with traditional and single-agent annotation techniques. Our framework is particularly effective at handling ambiguous queries, making it well-suited for deployment in production banking applications while showing strong generalization capabilities across different domains and languages. † † paperid: 123 1 Introduction Banking applications serve millions of customers daily, providing access to financial services and information. A critical aspect of these applications is the ability to quickly connect users with relevant information when they have questions. Frequently Asked Questions (FAQs) are a common mechanism for organizing and delivering this information. However, matching user queries to the most relevant FAQs remains challenging due to the diversity of user expressions, banking terminology, and the specific nature of financial inquiries. Traditional FAQ annotation approaches typically rely on a single technique, such as embedding-based retrieval, keyword matching, or direct large language model (LLM) inference. While each approach has strengths, they also have limitations: embedding models may miss semantic nuances, keyword matching can fail with synonyms or paraphrases, and direct LLM approaches may lack domain-specific knowledge or context. In this paper, we propose a multi-agent framework for FAQ annotation (MAFA). Our framework employs specialized agents to generate candidate FAQs, followed by a judge agent that reranks these candidates to produce a final, optimized ranking. This approach is inspired by recent advances in ensemble learning and LLM-based multi-agent systems introduced by Du et al. (2023) . Our framework achieves significant performance gains through strategic few-shot example selection tailored to individual agents. Rather than applying uniform examples across all agents, we curate distinct example sets that align with each agent’s unique capabilities and strengths. This targeted approach enhances performance across diverse query types while expanding the system’s coverage and adaptability. The multi-agent architecture, combined with a dedicated judge agent for intelligent reranking, delivers substantial improvements in FAQ mapping accuracy. Through comprehensive evaluation on both proprietary banking data and public benchmarks (LCQMC and FiQA), we demonstrate the framework’s effectiveness across varied domains and query patterns. Our implementation provides valuable insights for production deployment in banking environments, effectively handling challenges such as ambiguous queries and domain-specific terminology. These contributions establish a robust and practical methodology that advances FAQ annotation accuracy while maintaining the efficiency required for customer-facing applications. 2 Related work 2.1 FAQ retrieval systems Retrieval has been a focus of research for decades, evolving from simple keyword matching to sophisticated neural approaches. Early systems relied on information retrieval techniques like TF-IDF and BM25 Robertson and Zaragoza (2009) . With the advent of neural networks, embedding-based approaches became prevalent, including sentence-BERT Reimers and Gurevych (2019) and other transformer models that encode queries and FAQs into a shared semantic space. 2.2 Multi-agent systems Multi-agent systems have gained popularity with the rise of powerful LLMs. These systems typically involve multiple agents with specialized roles collaborating to solve complex reasoning tasks as showed by Du et al. (2023) ; Hegazy (2024) . Park et al. (2023) demonstrated the effectiveness of generative agents for collaborative problem-solving, while similarly, Wu et al. (2023) ; Li et al. (2023) presented approaches for creating autonomous agents that can collaborate through natural language communication. These works lay the groundwork for our multi-agent FAQ annotation system, though they did not specifically address the information retrieval domain. Our work draws inspiration from these approaches but focuses specifically on the FAQ annotation task in the banking domain, where precision and domain knowledge are crucial. 2.3 Retrieval-augmented generation (RAG) Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing large language models with external knowledge. Since its introduction by Lewis et al. (2020) , numerous advances have been made in improving retrieval quality, reranking strategies, and generation capabilities. Recent work by Lewis et al. (2020) has focused on enhancing RAG with more sophisticated retrieval mechanisms, hybrid search strategies Zhao et al. (2024) ; Kim et al. (2025) , and context-aware generation Su et al. (2023) . 2.4 Judging and reranking in information retrieval Reranking has been widely used in information retrieval to improve the relevance of search results as shown by Liu (2009) . Recent approaches leverage LLMs for reranking, as demonstrated by Nogueira et al. (2020) , who used BERT for document reranking. Our approach extends this concept to FAQ annotation, with a specialized judge agent that considers multiple sources of evidence. Recent work by Niu et al. (2025) introduced JudgeRank, a framework that leverages large language models for reasoning-intensive reranking tasks. This approach demonstrated the potential of using LLMs as judges to evaluate the quality of retrieved content, a concept we extend in our work. Similarly, Wang et al. (2023) showed how self-consistency can improve the reliability of LLM judgments, a principle we incorporate into our judge agent design. 2.5 Structured reasoning for large language models Recent research has shown that structured reasoning approaches can significantly improve LLM performance in complex tasks. Karov et al. (2025) introduced Attentive Reasoning Queries (ARQs), which guide LLMs through systematic reasoning steps using targeted, task-specific queries in a structured JSON format. This approach has been shown to outperform free-form Chain-of-Thought reasoning by directing the model’s attention to critical instructions and facilitating intermediate reasoning steps. ARQs counteract the ’lost in the middle’ phenomenon in autoregressive models, where information in the middle of long prompts tends to receive less attention than content at the beginning or end as shown by Liu et al. (2024) . By structuring the reasoning process with explicit queries in a JSON schema, ARQs ensure that important information remains in the model’s focus when it’s most needed for reasoning and decision-making. This approach has proven particularly effective for tasks requiring strict adherence to guidelines and prevention of hallucinations, which aligns well with our requirements for FAQ annotation. 3 Problem formulation The FAQ annotation problem can be formulated as follows: Given a user utterance u and a set of FAQs F=\{f_{1},f_{2},...,f_{n}\} , where each FAQ f_{i} consists of a question q_{i} and an answer a_{i} , the goal is to rank the FAQs in order of relevance to u