Adaptive Monitoring and Real‑World Evaluation of Agentic AI Systems

Manish A. Shukla · 2025

Read Adaptive Monitoring and Real‑World Evaluation of Agentic AI Systems by Manish A. Shukla (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00115.txt.

Introduction

Manish Shukla, working alone, sets out adaptive monitoring for agentic AI systems in real-world use in 2025. Such systems join language models with tools and self-guided planning, and they are moving into high-stakes fields. A prior paper gave a five-axis frame and early gauges such as goal drift, but no working method or proof from data.

Shukla reviews eighty-four papers and finds that skill gauges dominate, while human and cost axes are rare. He then formalizes a monitoring algorithm that scales mixed gauges to one range, sets moving thresholds per axis, and flags joint anomalies. The sequel turns a frame into a testable tool for live oversight. Live checks matter as agents gain real power.

Catalog record

CREATORManish A. Shukla
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00115

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Monitoring and Real‑World Evaluation of Agentic AI Systems Manish A. Shukla Abstract Agentic artificial intelligence (AI) — multi‑agent systems that combine large language models with external tools and autonomous planning — are rapidly transitioning from research laboratories into high‑stakes domains. Our earlier “Basic” paper introduced a five‑axis framework and proposed preliminary metrics such as goal drift and harm reduction but did not provide an algorithmic instantiation or empirical evidence. This “Advanced” sequel fills that gap. First, we revisit recent benchmarks and industrial deployments to show that technical metrics still dominate evaluations: a systematic review of 84 papers from 2023–2025 found that 83% report capability metrics while only 30% consider human‑centred or economic axes [ 2 ] . Second, we formalise an Adaptive Multi‑Dimensional Monitoring (AMDM) algorithm that normalises heterogeneous metrics, applies per‑axis exponentially weighted moving‑average thresholds and performs joint anomaly detection via the Mahalanobis distance [ 7 ] . Third, we conduct simulations and real‑world experiments. AMDM cuts anomaly‑detection latency from 12.3 s to 5.6 s on simulated goal drift and reduces false‑positive rates from 4.5% to 0.9% compared with static thresholds. We present a comparison table and ROC/PR curves, and we reanalyse case studies to surface missing metrics. Code, data and a reproducibility checklist accompany this paper to facilitate replication. Keywords: Agentic AI; Multi-agent systems; Evaluation framework; Adaptive Multi-Dimensional Monitoring (AMDM); Online anomaly detection; Goal drift; Safety and ethics. 1 Introduction Large language models (LLMs) underpin a new class of AI agents that can parse natural‑language instructions, call external tools and carry out multi‑step tasks. When multiple such agents are coordinated, they form agentic AI systems capable of planning, memory and delegation over extended horizons [ 3 ] . Existing evaluations largely focus on narrow technical metrics such as accuracy, latency and throughput, leaving sociotechnical dimensions like human trust, ethical compliance and economic sustainability under‑measured. A recent review of 84 papers reported that technical metrics dominate evaluations while human‑centred, safety and economic assessments are considered in only a minority of studies [ 2 ] . This measurement imbalance undermines claims of double‑digit productivity gains and multi‑trillion‑dollar economic potential [ 2 ] . The present work builds directly on our prior paper, “Evaluating Agentic AI Systems: A Balanced Framework for Performance, Robustness, Safety and Beyond” (henceforth “Basic”), which proposed a five‑axis evaluation framework and introduced goal‑drift and harm‑reduction metrics. “Basic” was primarily a conceptual position paper; it did not provide a concrete monitoring method or empirical evidence. Here we extend that foundation in three ways: (i) we examine recent benchmarks and deployments to quantify the persistent measurement imbalance and to motivate our algorithm; (ii) we develop Adaptive Multi‑Dimensional Monitoring (AMDM), a practical method for normalising metrics, applying per‑axis adaptive thresholds and performing joint anomaly detection; and (iii) we validate AMDM through simulations, real‑world logs and a reanalysis of case studies. These extensions transform the conceptual framework into an operational tool. Relation to our prior work. Basic introduced the five‑axis framework. This paper implements the framework with AMDM, presents ablations, ROC/PR curves and operational guidance, and reanalyses industrial case studies. Readers should consult Basic for the conceptual foundation; this paper focuses on the algorithmic instantiation and empirical validation. Contributions. In summary, this paper makes the following contributions: • We synthesise recent literature and industrial reports to demonstrate that technical metrics continue to dominate evaluations (83% of studies report capability metrics) while human‑centred, safety and economic axes are often ignored [ 2 ] . • We propose AMDM, a practical algorithm that normalises heterogeneous metrics, applies per‑axis adaptive thresholds via exponentially weighted moving averages, and performs joint anomaly detection using Mahalanobis distance. A concise pseudo‑code and complexity analysis accompany the method. • Through simulations and real‑world logs we show that AMDM cuts anomaly‑detection latency from 12.3 s to 5.6 s and lowers the false‑positive rate from 4.5% to 0.9% compared with static thresholds. We quantify sensitivity to hyperparameters ( \lambda , w , k ) and release our code and data to support reproducibility. • We reanalyse deployments in software modernisation, data quality and credit‑risk memo drafting. Although reported productivity gains range from 20% to 60% and credit‑turnaround times decrease by about 30% [ 4 ] , we highlight missing metrics such as developer trust, fairness and energy consumption. 2 Related Work and Persistent Gaps 2.1 Existing Metrics and Frameworks Early evaluations of agentic AI systems borrowed metrics from reinforcement learning and LLM benchmarks, emphasising task completion, latency and throughput. Subsequent works introduced metrics for robustness (e.g. resilience to noisy inputs), safety (toxicity, bias) and fairness, and human‑centred metrics such as user trust and transparency [ 5 ] . Economic and sustainability metrics (cost per interaction, productivity gains, carbon footprint) remain relatively rare [ 2 ] . Despite the proliferation of dashboards and benchmarks (e.g., MLAgentBench, PlanBench), most focus on capability and efficiency and omit integrated human or economic dimensions. 2.2 Measurement Imbalance in Practice As industry reports tout productivity gains, measurement imbalance can obscure real‑world risks. The McKinsey “Seizing the Agentic AI Advantage” playbook describes cases such as credit‑risk memo drafting and legacy modernisation where agentic AI systems purportedly deliver 20–60 % productivity improvements and 30 % faster credit decisions [ 4 ] . However, the public report primarily discusses capability and efficiency metrics, omitting fairness, user satisfaction and energy consumption. A systematic review found that only 15 % of studies jointly consider technical and human dimensions [ 2 ] . Case studies presented later in this paper reveal similar gaps: goal drift, hallucination and fairness violations remain undetected under static thresholds but are surfaced by our adaptive monitoring approach. 3 A Balanced Evaluation Framework To address measurement imbalance, our earlier paper defined five axes that together cover technical, human and economic dimensions. We briefly recap them here and refer the reader to Basic for details. 1. Capability & Efficiency : measures of task completion, latency and resource utilisation. 2. Robustness & Adaptability : resilience to noisy inputs, adversarial prompts and changing goals. 3. Safety & Ethics : avoidance of toxic or biased outputs and adherence to legal and ethical norms. 4. Human‑Centred Interaction : user satisfaction, trust and transparency. Instruments such as TrAAIT evaluate perceived credibility, reliability and value [ 5 ] . 5. Economic & Sustainability Impact : productivity gains, cost per outcome and carbon footprint. Figure 1 visualises the interdependencies among these dimensions: improvements in one may degrade another. A balanced evaluation therefore reports all axes and analyses trade‑offs rather than optimising a single metric. Figure 1: Interconnected evaluation dimensions for agentic AI systems. The five axes — capability&efficiency, robustness&adaptability, safetyðics, human‑centred interaction and economic&sustainability — are interdependent. Improvements in one dimension may affect others. This diagram is adapted from our earlier work. 4 Adaptive Multi‑Dimensional Monitoring The balanced framework defines what to measure. We now introduce AMDM, a concrete method that specifies how to measure and act on the five axes in real time. AMDM operates on streaming logs of agent actions, tool calls and user feedback, producing per‑axis alerts and joint anomaly signals. 4.1 Metric Normalisation Let m_{i}(t) denote the value of metric i at time t , where i indexes all metrics across the five axes. To compare heterogeneous metrics we compute a rolling

Read the complete text