When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors
Chenghao Yang, Yuning Zhang, Zhoufutu Wen, Tao Gong, Jiaheng Liu, Qi Chu, Nenghai Yu · 2026
Read When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors by Chenghao Yang, Yuning Zhang, Zhoufutu Wen, Tao Gong, Jiaheng Liu, Qi Chu, Nenghai Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21255.txt.
Introduction
Chenghao Yang and six colleagues measure how distillation makes AI agents behave alike, separating required task behaviors from freely chosen habits. Their two metrics track verbal alignment and tool-use habits as directed graphs. Across 18 models from 8 providers on tau-bench tasks, same-family pairs score higher on tool-use similarity than cross-family pairs, and Kimi-K2 even exceeds Anthropic's own Opus on node and dependency similarity. A controlled distillation test confirms the metrics catch teacher-specific convergence rather than general improvement.
Catalog record
| CREATOR | Chenghao Yang, Yuning Zhang, Zhoufutu Wen, Tao Gong, Jiaheng Liu, Qi Chu, Nenghai Yu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69799 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21255
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors Chenghao Yang, Yuning Zhang, Zhoufutu Wen, Tao Gong, Jiaheng Liu, Qi Chu, Nenghai Yu Abstract Model distillation is a primary driver behind the rapid progress of LLM agents, yet it often leads to behavioral homogenization. Many emerging agents share nearly identical reasoning steps and failure modes, suggesting they may be distilled echoes of a few dominant teachers. Existing metrics, however, fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model’s autonomous preferences. We propose two complementary metrics to isolate non-mandatory behavioral patterns: Response Pattern Similarity (RPS) for verbal alignment and Action Graph Similarity (AGS) for tool-use habits modeled as directed graphs. Evaluating 18 models from 8 providers on τ \tau -Bench and τ 2 \tau^{2} -Bench against Claude Sonnet 4.5 (thinking), we find that within-family model pairs score 5.9 pp higher in AGS than cross-family pairs, and that Kimi-K2 (thinking) reaches 82.6% S node S_{\text{node}} and 94.7% S dep S_{\text{dep}} , exceeding Anthropic’s own Opus 4.1. A controlled distillation experiment further confirms that AGS distinguishes teacher-specific convergence from general improvement. RPS and AGS capture distinct behavioral dimensions (Pearson r r = 0.491), providing complementary diagnostic signals for behavioral convergence in the agent ecosystem. Our code is available at https://github.com/Syuchin/AgentEcho . uto counter, number within=section]Prompt[2][]colback=white, colframe=cyan, width=arc=5mm, boxrule=0.8mm, title=#2, breakable, fonttitle=, listing only, listing options= basicstyle= , breaklines=true, breakautoindent=false, columns=flexible, , #1 † † footnotetext: \dagger Corresponding authors. (a) Example trajectories on an item exchange task. (b) RPS vs. AGS across non-Anthropic models. Figure 1: Behavioral similarity to Claude Sonnet 4.5 (thinking). (a) shows example trajectories from Kimi-K2, Claude, and GPT-5 on an item exchange task. Kimi-K2 and Claude share similar response phrasing and optional tool choices, while GPT-5 adopts a different style. (b) plots RPS against AGS for all non-Anthropic models. Kimi-K2 (thinking) occupies the top-right corner, closest to the reference model. 1 Introduction The current "Cambrian explosion" of high-performing LLM agents often carries a persistent sense of déjà vu. Despite their diverse origins, many emerging agents exhibit a considerably aligned behavior: they share nearly identical reasoning steps, redundant tool-calling habits, and even the same failure modes ( 43 ; 28 ; 23 ) . This suggests that instead of independent breakthroughs, these models may be distilled echoes of a few dominant teachers. Such pervasive mimicry leads to a convergence of "bad habits" across theoretically independent models ( 35 ; 6 ) . For instance, rather than optimizing for efficiency, many agents mirror the teacher model’s verbose reasoning and redundant tool-calling patterns, such as trial-and-error with every available tool, even when the solution is obvious ( 27 ; 47 ) . This collective alignment means that the ecosystem lacks actual robustness; different models no longer provide independent verification but instead fail in the exact same way ( 16 ; 42 ) . Quantifying this distillation-induced alignment is essential for ensuring ecosystem transparency, but existing methods fall short in the complex, multi-step agent landscape. Current metrics primarily focus on response-level similarity in static dialogues, which fails to capture the dynamic nature of tool-use trajectories ( 25 ) . More critically, they struggle to distinguish between mandatory behaviors related to actions strictly required for task success and non-mandatory behaviors, which reflect a model’s autonomous preferences. Without isolating these "behavioral degrees of freedom," it is impossible to determine whether two models converge because there is only one correct path, or because one is blindly shadowing the other. To bridge this gap, we propose a systematic framework to quantify agent distillation by isolating non-mandatory behavioral patterns. Our approach introduces two complementary metrics: Response Pattern Similarity (RPS) and Action Graph Similarity (AGS). RPS measures verbal similarity by segmenting trajectories into five canonical stages (authentication, elicitation, execution, verification, notification) and scoring similarity along style, structure, and alignment dimensions. AGS measures action-level similarity through three sub-metrics. S_{\text{node}} captures optional tool agreement : for a flight cancellation task, all models must call cancel_reservation , but some additionally call get_reservation_details to double-check. When two models share such optional choices, they likely share training signals. S_{\text{seq}} captures sequential habits such as post-write verification, pre-modification confirmation, and error retry patterns. S_{\text{dep}} captures dependency patterns such as output reuse rate and tool-chaining depth. By isolating these non-mandatory behaviors from task-dictated actions, our metrics reveal stylistic and structural alignment that would otherwise be masked by shared correctness. We evaluate 18 models from 8 major providers on \tau -Bench and \tau^{2} -Bench, using Claude Sonnet 4.5 (thinking) as the reference "oracle". Our experiments reveal several key findings. (1) Anthropic models exhibit strong internal consistency with RPS scores above 3.8, consistent with shared training pipelines. (2) Kimi-K2 (thinking) shows elevated similarity on both metrics, achieving the highest AGS at 82.7% among non-Anthropic models, with S_{\text{node}} at 82.6% and S_{\text{dep}} at 94.7%. (3) RPS and AGS capture distinct behavioral dimensions, providing complementary signals for distillation. (4) A controlled distillation experiment shows that AGS rises toward the teacher and drops toward a non-teacher control, while baseline metrics rise toward both. Our findings provide empirical grounding for the initial déjà vu. While some models maintain high behavioral diversity, others exhibit elevated similarity with the reference model across both verbal and structural dimensions, even in scenarios where multiple alternative paths exist. Our main contributions are summarized as follows. • We propose the first framework for tool-use agent distillation quantification, disentangling mandatory and non-mandatory behaviors in tool-use trajectories. • We introduce RPS and AGS, providing interpretable metrics for verbal and action-level behavioral similarity. • We evaluate 18 models from 8 providers, analyzing behavioral convergence patterns in the current agent ecosystem. 2 Related Work Knowledge Distillation for LLM. Knowledge distillation transfers knowledge from a large teacher model to a smaller student model 18 and has been widely applied to compress pre-trained models, such as BERT 41 . For large language models, it is typically categorized into white-box 24 and black-box distillation 44 ; 7 . White-box methods require access to intermediate layers or logits 22 ; 45 , whereas black-box approaches use teacher-generated sequences 1 ; 52 ; 19 , allowing for distillation from proprietary APIs or arbitrary architectures. However, empirical studies show that distillation can cause homogenization problems in LLMs 25 . Data Contamination. Data contamination, or benchmark leakage 29 ; 48 , refers to unintentional inclusion of test and benchmark data in training corpora, posing challenges to reliable LLM evaluation 39 . Such overlaps inflate performance and undermine fair comparisons 29 . Solutions include contamination detection 13 ; 12 and dynamic evaluation 50 . Recent methods include distributional memorization 46 using task-gram language models on semantically related n-grams, and kernel divergence score from embedding kernel similarity matrices 8 . Tool-use Benchmark. Tool use is a key capability of LLM agents, spurring multiple benchmarks for evaluation. Early benchmarks like API-Bank 26 and Gorilla 34 target tool selection and real API calling in large-scale tool sets. ToolBench 36 extends this to multi-step interactions. BFCL 33 focuses on cross-domain function-calling accuracy. TRAJECT-Bench 17 introduces trajectory-aware metrics evaluating selection, parameterization, ordering, and dependencies. Domain-specific benchmarks have also emerged, including finance 20 and medicine 21 . In conversational agent settings, \tau -Bench 49 and \tau^{2} -Bench 4