AgentPulse: A Continuous Multi-Signal Framework for Evaluating AI Agents in Deployment

Yuxuan Gao, Megan Wang, Yi Ling Yu, [0.5em] University of Pennsylvania · 2026

Read AgentPulse: A Continuous Multi-Signal Framework for Evaluating AI Agents in Deployment by Yuxuan Gao, Megan Wang, Yi Ling Yu, [0.5em] University of Pennsylvania (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24038.txt.

Introduction

Gao, Wang, and Yu introduce AgentPulse, a live evaluation framework that scores 50 agents across 10 workload types. Four factors, benchmark results, adoption signals, community feeling, and ecosystem health, draw on 18 real-time signals from code hosts, registries, marketplaces, social sites, and leaderboards. A test free of code-host signals still predicts outside adoption measures, while benchmark-only and full rankings nearly disagree on a small subset. The authors stress it is a method for surfacing deployment signals, not a final ranking.

Catalog record

CREATORYuxuan Gao, Megan Wang, Yi Ling Yu, [0.5em] University of Pennsylvania
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56103
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AgentPulse: A Continuous Multi-Signal Framework for Evaluating AI Agents in Deployment Yuxuan Gao, Megan Wang, Yi Ling Yu, [0.5em] University of Pennsylvania Abstract Static benchmarks measure what AI agents can do at a fixed point in time but not how they are adopted, maintained, or experienced in deployment. We introduce AgentPulse , a continuous evaluation framework scoring 50 agents across 10 workload categories along four factors — Benchmark Performance, Adoption Signals, Community Sentiment, and Ecosystem Health — aggregated from 18 real-time signals across GitHub, package registries, IDE marketplaces, social platforms, and benchmark leaderboards. Three analyses ground the framework. The four factors capture largely complementary information ( n = 50 n{=}50 ; ρ max = 0.61 \rho_{\max}{=}0.61 for Adoption–Ecosystem, all others | ρ | ≤ 0.37 |\rho|{\leq}0.37 ). A circularity-controlled test ( n = 35 n{=}35 ) shows the Benchmark+Sentiment sub-composite — which contains no GitHub-derived signals — predicts external adoption proxies it does not aggregate: GitHub stars ( ρ s = 0.52 \rho_{s}{=}0.52 , p < 0.01 p{<}0.01 ) and Stack Overflow question volume ( ρ s = 0.49 \rho_{s}{=}0.49 , p < 0.01 p{<}0.01 ), with VS Code installs ( ρ s = 0.44 \rho_{s}{=}0.44 , p < 0.05 p{<}0.05 ) reported as illustrative given that only 11 of 35 agents have non-zero installs. On the n = 11 n{=}11 subset with published SWE-bench scores, composite and benchmark-only rankings are nearly uncorrelated ( ρ s = 0.25 \rho_{s}{=}0.25 ; 9 of 11 agents shift by ≥ 2 \geq 2 ranks), driven by a strong negative Adoption–Capability correlation among closed-source high-capability agents within this subset — which is precisely why we rest the framework’s validity claim on the broader n = 35 n{=}35 test rather than the SWE-bench overlap. AgentPulse surfaces deployment signal absent from benchmarks; it is a methodology, not a ground-truth ranking. The framework, all collected signals, scoring outputs, and evaluation harness are released under CC BY 4.0. 1 Introduction AI agents — systems that combine language model reasoning with tool use, code execution, and multi-step planning — have moved from research prototypes to production tools used by millions of developers. Existing evaluation, however, remains anchored in static benchmarks: SWE-bench [ 6 ] , GAIA [ 10 ] , WebArena [ 16 ] , AgentBench [ 8 ] , and HumanEval+ [ 3 ] measure capability at fixed points in time. These benchmarks are essential but incomplete: they cannot capture how agents evolve through frequent updates, how reliability varies under real-world load, or how developer experience changes as workflows adapt to these tools. Treating evaluation as a scientific object of study in its own right, this paper investigates evaluation methodology for deployed AI agents. We ask: does a deployment-aware composite of public signals capture information benchmarks miss, and can such a composite be validated independently of the signals it aggregates? Our contribution is a measurement framework with three validation analyses, not a leaderboard. We motivate this question through three concrete limitations of current agent evaluation: Limitation 1: Static snapshots. Existing benchmarks measure agents at a point in time. They cannot capture the rapid release cadence of agentic systems, in which capabilities, integrations, and reliability shift on weekly timescales. Limitation 2: Capability \neq adoption. An agent’s task-completion score does not predict which tools developers actually choose. Adoption depends on integration quality, pricing, reliability, documentation, and community — dimensions invisible to benchmarks. Limitation 3: No cross-tool comparability. Existing benchmarks evaluate narrow capability slices. There is no unified framework that compares a coding copilot, an autonomous SWE agent, and a multi-agent framework along their respective strengths while accounting for the fact that these tools serve different workflows. Contributions. 1. A four-factor evaluation framework — Benchmark Performance, Adoption Signals, Community Sentiment, and Ecosystem Health — with explicit design rationale (Section 3 ). 2. An 18-signal real-time data pipeline aggregating across GitHub, package registries, IDE marketplaces, social platforms (Bluesky, Reddit, Hacker News, Stack Overflow, Mastodon), and five benchmark leaderboards, evaluating 50 agents across 10 workload categories (Section 4 ). 3. Three validation analyses: factor independence ( n{=}50 ); circularity-controlled cross-factor predictive validity ( n{=}35 , p{<}0.01 ); and ranking divergence ( n{=}11 , framed as descriptive given the small sample) (Section 5 ). 4. A factor ablation that transparently characterizes the composite’s behavior, including a structural tension on the n{=}11 SWE-bench subset that we interpret rather than paper over (Section 6 ). 5. A public release of the framework, all collected signals, scored texts, and an evaluation harness designed for continuous, community-extensible agent evaluation (Section 8 ). 2 Related Work Agent benchmarks. SWE-bench [ 6 ] evaluates coding agents on real GitHub issue resolution; GAIA [ 10 ] tests general assistant capability across browsing, reasoning, and tool use; WebArena [ 16 ] measures browser automation; AgentBench [ 8 ] provides a multi-environment evaluation suite; and HumanEval+ [ 3 ] extends code generation evaluation. These benchmarks measure what agents can do at a point in time, not how they are adopted and experienced over time. AgentPulse is complementary: it operates on top of these benchmarks as one of four factors and adds dimensions they do not capture. Holistic and continuous evaluation. HELM [ 7 ] introduced multi-metric evaluation for language models. LMSYS Chatbot Arena [ 4 , 15 ] pioneered continuous, preference-based LLM evaluation through pairwise human comparisons — providing a clear external ground truth that single-metric benchmarks lack. AgentPulse extends the holistic and continuous evaluation paradigm to agent ecosystems but takes a different validation approach: rather than soliciting human preferences, we ground the composite by testing that benchmark-and-sentiment factors alone predict observable adoption. Multi-signal aggregation. Social-media sentiment [ 2 ] , domain-specific NLP [ 1 ] , and aspect-based sentiment analysis [ 11 ] are established techniques in adjacent fields. We adapt these to agent evaluation with agent-specific lexicons and combine them with adoption and ecosystem health metrics that have no analog in prior agent benchmarks. Critiques of benchmark methodology [ 12 ] highlight precisely the kinds of blind spots — adoption, deployment context, real-world signal — that AgentPulse is designed to address. 3 The AgentPulse Framework AgentPulse evaluates each agent through a composite of four factors: \mathrm{AP}(a)=w_{B}\cdot B(a)+w_{A}\cdot A(a)+w_{S}\cdot S(a)+w_{E}\cdot E(a) (1) where \mathrm{AP}(a) denotes the composite (calligraphic) and S(a) the sentiment factor defined below, with w_{B}{=}0.35 , w_{A}{=}0.25 , w_{S}{=}0.20 , w_{E}{=}0.20 . The allocation is a principled prior, not an empirical optimum: benchmark capability receives the largest weight because task completion is the most direct measure of whether an agent does what it claims; the remaining 65\% captures dimensions benchmarks alone cannot measure. The framework supports custom weightings, and we test robustness through sensitivity (Section 5.4 ) and ablation (Section 6 ). Factor 1: Benchmark Performance ( B ). The average of all available published benchmark scores, normalized to [0,1] : B(a)=\frac{1}{|\mathcal{B}_{a}|}\sum_{b\in\mathcal{B}_{a}}\frac{s_{b}(a)}{100} (2) where \mathcal{B}_{a} is the set of benchmarks for which agent a has a published score among SWE-bench Verified, GAIA, WebArena, HumanEval+, and TAU-bench [ 14 ] . Agents without published benchmarks receive a neutral prior of 0.5 rather than the cross-sectional mean, ensuring the prior remains stable as new agents enter the registry. Factor 2: Adoption Signals ( A ). Log-normalized metrics blending code hosting, package distribution,

Read the complete text