Performant LLM Agentic Framework for Conversational AI 1 Equal Contribution.

Alex Casella 1, Wayne Wang 1 · 2025

Read Performant LLM Agentic Framework for Conversational AI 1 Equal Contribution. by Alex Casella 1, Wayne Wang 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06410.txt.

Introduction

Alex Casella and Wayne Wang present a framework that keeps voice AI agents on the right path through branching conversation graphs. Businesses now route callers through webs of decision nodes. Language models steering those webs can pick wrong nodes or drown in context and invent steps.

Their system pairs model reasoning with a vector score that ranks the next node mathematically. It balances strict path-following against smart jumps when callers go off script. Tests beat baseline methods on accuracy and speed together. Call centers and voice assistants are the market. Reliable graph walking is what turns a demo agent into a deployable one. The catalog title carries an equal-contribution footnote as junk text.

Catalog record

CREATORAlex Casella 1, Wayne Wang 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06410

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Performant LLM Agentic Framework for Conversational AI 1 Equal Contribution. Alex Casella 1, Wayne Wang 1 Abstract The rise of Agentic applications and automation in the Voice AI industry has led to an increased reliance on Large Language Models (LLMs) to navigate graph-based logic workflows composed of nodes and edges. However, existing methods face challenges such as alignment errors in complex workflows and hallucinations caused by excessive context size. To address these limitations, we introduce the Performant Agentic Framework (PAF), a novel system that assists LLMs in selecting appropriate nodes and executing actions in order when traversing complex graphs. PAF combines LLM-based reasoning with a mathematically grounded vector scoring mechanism, achieving both higher accuracy and reduced latency. Our approach dynamically balances strict adherence to predefined paths with flexible node jumps to handle various user inputs efficiently. Experiments demonstrate that PAF significantly outperforms baseline methods, paving the way for scalable, real-time Conversational AI systems in complex business environments. I Introduction Graph-based workflows are central to numerous business processes across industries such as education, legal, healthcare, and customer support. These workflows represent decision-making steps as nodes, and connections between them as edges. The rise of Conversational AI within these spaces introduces new challenges. Autonomous agents, powered by large language models (LLMs), are increasingly being used to navigate these workflows, enabling the automation of complex business processes ( Zhuge et al. 2023 ) . Each node in the workflow contains specific instructions or prompts that guide the agent’s speech generation and certain actions to trigger. Nodes can be classified into several types, including Start Nodes, which define the root and entry point of a workflow; End Nodes, which signal the termination of the workflow; and generic Nodes, which serve as intermediate decision points containing speech instructions for the LLM to converse with users in predefined ways. Additionally, Transfer Nodes in Conversational AI workflows allow for transitioning the conversation to another autonomous or human agent. Edges between nodes may include logical conditions that dictate the agent’s transitions, ensuring workflows are executed accurately. Fig. 1: Example illustration of an Agentic workflow for a healthcare call center use case, where the Agent needs to route calls based on different conditions. Figure 1 illustrates how tasks such as determining health care eligibility can be broken down into nodes, edges, and conditions. For example, a healthcare provider might use such a workflow to efficiently filter out patients without the required insurance, reducing the burden on human agents. However, workflows like these can rapidly grow in complexity. As shown in Figure 2 , adding just a few additional conditions to the conversation flow can drastically increase the number of nodes and edges, making the workflow more difficult to manage and execute effectively. Fig. 2: Example illustration of an Agentic workflow for an internet service company helping callers troubleshoot connection issues. This workflow demonstrates how a more complex use case can have more conditions, nodes, and edges. Although LLMs such as GPT and LLAMA are built on autoregressive decoder-based transformer architectures optimized for natural language generation, they are not inherently designed to handle structured, multi-step processes with extensive context ( Qiu and Jin 2024 ; Shi et al. 2023 ) . Existing approaches have been to add a planning phase, where the LLM would take time to orchestrate the action, and then proceed to the generation tasks ( Valmeekam et al. 2023 ; Zhou et al. 2024 ) . However, this approach is not optimal to the Conversational AI use case, as it would increase the overall latency by doubling the number of queries needed. Tasks such as managing end-to-end customer service requests with non-standard return policies, performing outbound sales calls that involve dynamic CRM updates, or redirecting users to appropriate departments after a sequence of filtering questions require precision, alignment, and low-latency responses. These limitations force businesses to oversimplify workflows, sacrificing accuracy and operational efficiency—an outcome that is contrary to their objectives. The challenges inherent in adapting LLMs to graph-based workflows underscore the need for new approaches that can accurately and efficiently execute workflows while respecting real-world constraints such as latency. While adding more reasoning steps could theoretically improve accuracy, such methods are impractical for Conversational AI applications where rapid response times are critical. To address these challenges in the current Conversational AI space, this paper introduces the Performant Agentic Framework (PAF) , a novel solution for efficient graph traversal that balances accuracy and latency in real-world applications. By leveraging both traditional decision-making logic and mathematical methods for next-node selection, PAF enables agents to execute workflows with greater precision and speed. Our experiments demonstrate that PAF significantly outperforms baseline and traditional methods in both accuracy and latency, as evidenced by higher alignment scores and reduced response times. II Related Work The reliance on LLM-based systems to execute graph-based workflows has seen significant research attention, particularly in developing frameworks that aim to balance accuracy, latency, and alignment with predefined workflows. Below, we discuss prominent related works and their limitations. Agentic Framework Serving as examples, LangChain ( LangChain 2023 ) and LangGraph ( LangGraph 2023 ) streamline graph-based workflows by utilizing function calls and prompt chaining. While effective for simple tasks, their reliance on keyword-based triggers often results in alignment errors, especially in workflows with hundreds or thousands of nodes. These frameworks lack robustness for real-world applications where actions must be dynamically triggered at various points in conversations. Furthermore, their reliance on LLM-generated triggers leads to unreliability in critical workflows, where adherence to predefined paths is essential for compliance and business logic ( LangChain 2023 ; LangGraph 2023 ) . Additionally, limitations in LLM context windows further exacerbate their inefficiency in retaining relevant information across extended workflows, introducing hallucinations and context drift during execution ( Dong and Qian 2024 ) . Conversational AI Conversational AI has been a key focus for Natural Language Processing. The existing Conversational AI solutions emphasize the need for multi-modality, guardrails, and advanced tuning to enhance dialogue quality ( Dong et al. 2023 ) . Prior approaches to the Voice AI space have been proven to work in a sandbox conversational setting ( James et al. 2024 ) , but lack the consistency and accuracy required for production use. As suggested, LLMs miss certain abilities to maintain performance in a dynamic conversational setting, unable to handle numerous tasks conditionally while reducing hallucinations and staying within context ( Gill and Kaur 2023 ; Dong et al. 2023 ; Dong and Qian 2024 ) . MetaGPT and SOP Translation MetaGPT leverages Standardized Operating Procedures (SOPs) to structure workflows, enabling agents to replicate domain-specific expertise. However, its reliance on iterative planning and validation increases latency, making it unsuitable for real-time applications. For example, as noted in ( Gao et al. 2023 ) , the planning phase requires additional LLM calls, which adds computational overhead. While MetaGPT is effective for SOP alignment, it struggles with unusual user inputs and extended workflows, leading to significant context drift. Its dependence on domain-specific fine-tuning also hinders generalizability, limiting its use in broader applications ( Gao et al. 2023 ; Wang and Liu 2024 ) . Comparison and Our Contributions Existing frameworks have made valuable contributions but are hindered by issues such as context drift,

Read the complete text