AgentCollabBench : Diagnosing When Good Agents Make Bad Collaborators
Aritra Mazumder, Shubhashis Roy Dipta, Nusrat Jahan Lia, Tanzila Khan, Kainat Raisa Hossain, Nehaa Shri, Shubhrangshu Debsarkar, Humayra Tasnim, Gour Gupal Talukder Shawon, Debjoty Mitra, Sumaiya Ahmed Rani, Al Jami Islam Anik, Al Nafeu Khan · 2026
Read AgentCollabBench : Diagnosing When Good Agents Make Bad Collaborators by Aritra Mazumder, Shubhashis Roy Dipta, Nusrat Jahan Lia, Tanzila Khan, Kainat Raisa Hossain, Nehaa Shri, Shubhrangshu Debsarkar, Humayra Tasnim, Gour Gupal Talukder Shawon, Debjoty Mitra, Sumaiya Ahmed Rani, Al Jami Islam Anik, Al Nafeu Khan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08647.txt.
Introduction
Aritra Mazumder and twelve coauthors build AgentCollabBench, a diagnostic test for failures that hide inside multi agent teamwork. A final answer can look right while an agent quietly dropped a constraint upstream, and outcome only grading never sees the damage. Their nine hundred human checked tasks span software, operations, and data work, each isolating one of four risks.
The risks cover constraints fading under peer pressure, falsehoods spreading by consensus, information leaking between tasks, and marked data failing to reach the end. Four leading models show distinct weakness profiles, and network shape explains much of the loss, especially where branches merge. Multi agent design emerges as a structural reliability problem, not just a model quality one.
Catalog record
| CREATOR | Aritra Mazumder, Shubhashis Roy Dipta, Nusrat Jahan Lia, Tanzila Khan, Kainat Raisa Hossain, Nehaa Shri, Shubhrangshu Debsarkar, Humayra Tasnim, Gour Gupal Talukder Shawon, Debjoty Mitra, Sumaiya Ahmed Rani, Al Jami Islam Anik, Al Nafeu Khan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 109 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08647
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AgentCollabBench : Diagnosing When Good Agents Make Bad Collaborators Aritra Mazumder, Shubhashis Roy Dipta, Nusrat Jahan Lia, Tanzila Khan, Kainat Raisa Hossain, Nehaa Shri, Shubhrangshu Debsarkar, Humayra Tasnim, Gour Gupal Talukder Shawon, Debjoty Mitra, Sumaiya Ahmed Rani, Al Jami Islam Anik, Al Nafeu Khan Abstract Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint, the system’s final output may look correct even though the reasoning chain was quietly corrupted, and existing outcome-based evaluations are blind to such multi-hop process failures. To make these vulnerabilities measurable before deployment, we introduce AgentCollabBench , a diagnostic benchmark of 900 human-validated tasks spanning software engineering, DevOps, and data engineering. Each task isolates one of four behavioral risks: instruction decay (does a constraint survive peer pressure?), false-belief contagion (does a falsehood spread through consensus?), context leakage (does information bleed between tasks?), and tracer durability (does marked data reach the final agent?). Evaluating four modern LLMs (GPT 4.1 mini, Gemini 2.5 Flash Lite, Qwen-3.5-35B-A3B, and Llama 3.1 8B Instruct), we expose model-specific vulnerability profiles invisible to outcome-only evaluation; Qwen-3.5-35B-A3B, for example, leads on tracer durability and instruction stability, while GPT 4.1 mini leads on leakage containment and false-belief resistance. Beyond per-model differences, communication topology emerges as a primary risk factor that explains 7-40% of the variance in multi-hop information survival. The effect traces to a synthesis bottleneck specific to converging-DAG nodes: an agent weighing competing parent inputs discards constraints carried by a minority branch, a bottleneck structurally absent from linear chains. AgentCollabBench demonstrates that suboptimal topology can silently erase the safeguards of highly capable models, arguing that multi-agent reliability is fundamentally a structural problem and that scaling model intelligence alone is no substitute for architecture. 1 1 1 Evaluation Framework: https://github.com/aritra741/AgentCollabBench 1 Introduction Consider a three-agent DevOps pipeline: a Configuration Manager, a Security Auditor, and a Deployment Engineer synthesize their inputs into a final Kubernetes manifest. The Configuration Manager relays a critical constraint: a verified container image. The Security Auditor, working from a different context, specifies a different image. The Deployment Engineer produces a complete, valid, deployment-ready manifest. It passes every outcome check. But the verified image is nowhere in the output. This is not just an edge case: across four modern LLMs evaluated on 900 controlled tasks, communication topology alone explains 7-40% of the variance in whether such constraints survive multi-hop forwarding. The effect is comparable in magnitude to the choice of backbone model. The verified image was displaced at the convergence node by the conflicting input, which arrived with equal syntactic authority. Outcome-based benchmarks which ask only whether the correct task completion is achieved, cannot detect it. Outcome accuracy, while a natural starting point for a single-agent system, misses the failures that matter most once a system is deployed. Recent multi-agent work has begun to look beyond outcome accuracy: GEMMAS [ 19 ] introduces graph-based metrics, MultiAgentBench [ 49 ] proposes milestone-based KPIs, and AgentsNet [ 13 ] evaluates coordination across topologies. None provides a quantitative, controllable instrument for measuring whether agents maintain hard constraints under peer pushback, whether critical information survives multi-hop forwarding, whether false beliefs spread through team consensus, or whether private context leaks across task boundaries (privacy-focused benchmarks [ 45 , 48 ] address this only in single-agent or single-session settings). A multi-agent system can score well on every existing benchmark while failing on all four of these deployment-critical dimensions. In our experiments, Llama 3.1 8B Instruct violates injected constraints 10.1% of the time and leaks cross-task context most often (4.9%); Qwen-3.5-35B-A3B preserves tracers near-ceiling (RTD 94%) but still spreads injected false beliefs 20.7% of the time; GPT 4.1 mini and Gemini 2.5 Flash Lite exhibit distinct intermediate profiles that outcome-based evaluation does not detect. In every case the team produces a syntactically complete, task-relevant response. The converging-DAG pattern from the vignette also generalizes: across all four evaluated models, converging topologies are associated with substantially higher constraint loss than linear chains. Two pipelines can produce indistinguishable outputs and yet carry opposite behavioral risk profiles, depending on how their agents are connected. AgentCollabBench is a diagnostic benchmark built to make these vulnerabilities measurable before deployment. Like a static analyzer that flags unsafe code patterns without requiring a crash, its value lies in surfacing specific, auditable risk signals prior to production. Our experiments span three enterprise domains (software engineering, DevOps, and data engineering), five communication topologies (linear chain, branching tree, converging DAG, fully connected, and custom graph), and three complexity tiers, four metrics and all human-validated tasks. The behavioral profiles exposed by the benchmark are not explained by general capability. Models that perform strongly on standard capability benchmarks do not consistently perform well across our behavioral metrics ( table 2 ). 2 2 2 Representative failure modes and converging-DAG trace audits are in appendices C and H . AgentCollabBench makes three contributions: 1. A balanced 900-task benchmark spanning software engineering, DevOps, and data engineering; easy, medium, and hard structural complexity tiers; and five communication topologies (linear chain, branching tree, converging DAG, fully connected, and custom graph). The topology dimension is not cosmetic: topology effects on RTD are comparable in magnitude to backbone model choice, with replicated \eta^{2}=0.07 - 0.40 across balanced task slices. 2. Four diagnostic metrics : Instruction Decay Rate (IDR) for constraint decay under peer pressure, Radioactive Tracer Durability (RTD) for information loss across hops, Consensus Pollution Rate (CPR) for false-belief propagation, and Cross-task Leakage Containment (CLC) for private context leakage across task boundaries. 3. A reusable evaluation framework for diagnosing behavioral failures in any multi-agent LLM system, packaging the four metrics, controlled artifact injection, perturbation-ladder construct validation, and a judge–backbone separation protocol. All task definitions, injection artifacts, judge prompts, and scoring code are released, so practitioners can score their own pipelines against the same diagnostic axes before deployment. 2 Related Work 2.1 Task accuracy benchmarks Agent evaluation has progressed from single-agent outcome benchmarks (AgentBench [ 24 ] , GAIA [ 28 ] , and SWE-bench [ 18 ] ) toward collaborative settings. MultiAgentBench [ 49 ] extends evaluation into collaborative settings through milestone-based KPIs, though its coordination signals remain qualitative. The MAST taxonomy [ 2 ] provides fine-grained failure categorization through systematic human analysis of 1,642 execution traces across seven frameworks and identifies fourteen failure modes. Despite its analytical depth, MAST scales annotations using an LLM-as-a-Judge [ 4 ] paradigm that, while achieving strong human agreement, produces qualitative labels assigned after task completion. AgentCollabBench addresses such lack of quantitative, temporally grounded signals required to trace how failures emerge and evolve during interaction. 2.2 Process-level metrics and failure dynamics Evaluation strategies have started to shift from task completion alone toward internal dynamics. GEMMAS [ 19 ] derives graph-based metrics over agent communication structure; Collab-Overcooked [ 42 ] measures trajectory efficiency as successful completions per agent step. Both treat the trajectory as given and neither probes whether the system behaves correctly under conditions that expose specific failure modes. Complementary evidence shows that agents frequently transition from correct to incorrect answers due to peer influence [ 44 ] and that multi-agent deliberation is degraded by sycophancy [ 34 ] : agents abandon correct reasoning to mimic their peers rather than critically evaluating flawed