Trace Sampling 2.0: Code Knowledge Enhanced Span-level Sampling for Distributed Tracing Conference: ; Apr 14 – 21, 2026; Rio De Janeiro, Brazil

Yulun Wu, Guangba Yu, Zhihan Jiang, Yichen Li, Michael R. Lyu · 2025

Read Trace Sampling 2.0: Code Knowledge Enhanced Span-level Sampling for Distributed Tracing Conference: ; Apr 14 – 21, 2026; Rio De Janeiro, Brazil by Yulun Wu, Guangba Yu, Zhihan Jiang, Yichen Li, Michael R. Lyu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13852.txt.

Introduction

Yulun Wu and four co-authors rethink distributed tracing in a 2025 systems paper. Microservice applications record traces of each request's path, but the volume overwhelms storage, so samplers keep only selected traces, usually the abnormal ones, throwing away the normal traces needed for comparison.

Their Trace Sampling 2.0 works at the span level, the individual operations inside a trace, keeping every trace while shrinking each one. The implementation, Autoscope, uses static code analysis to learn the execution logic and preserve the spans that matter without breaking the trace structure. On two open-source microservices it cut trace size by about four fifths while keeping nearly all faulty spans, and it improved root cause analysis accuracy as well.

Catalog record

CREATORYulun Wu, Guangba Yu, Zhihan Jiang, Yichen Li, Michael R. Lyu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-13852

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trace Sampling 2.0: Code Knowledge Enhanced Span-level Sampling for Distributed Tracing Conference: ; Apr 14 – 21, 2026; Rio De Janeiro, Brazil Yulun Wu, Guangba Yu, Zhihan Jiang, Yichen Li, Michael R. Lyu Abstract . Distributed tracing is an essential diagnostic tool in microservice systems, but the sheer volume of traces places a significant burden on backend storage. A common approach to mitigating this issue is trace sampling, which selectively retains traces based on specific criteria, often preserving only anomalous ones. However, this method frequently discards valuable information, including normal traces that are essential for comparative analysis. To address this limitation, we introduce Trace Sampling 2.0, which operates at the span level while maintaining trace structure consistency. This approach allows for the retention of all traces while significantly reducing storage overhead. Based on this concept, we design and implement Autoscope, a span-level sampling method that leverages static analysis to extract execution logic, ensuring that critical spans are preserved without compromising structural integrity. We evaluated Autoscope on two open-source microservices. Our results show that it reduces trace size by 81.2% while maintaining 98.1% faulty span coverage—outperforming existing trace-level sampling methods. Furthermore, we demonstrate its effectiveness in root cause analysis, achieving an average improvement of 8.3%. These findings indicate that Autoscope can significantly enhance observability and storage efficiency in microservices, offering a robust solution for performance monitoring. 1. Introduction Modern software architectures have evolved into distributed microservice systems ( Newman 2021 ; Zhou et al. 2021 ) , making distributed tracing an essential observability tool for understanding application behavior and performance at scale ( Sigelman et al. 2010 ; Huang et al. 2025 ; Fonseca et al. 2007 ; Liu et al. 2023 ) . As shown the Trace-1 in Fig. 1 , by capturing fine-grained spans along request paths across services, traces create a comprehensive execution path that includes service transitions, time distributions, and context propagation. These tracing systems enable Site Reliability Engineers (SREs) to effectively profile system performance ( Zhang et al. 2022c ; Huang and Zhu 2021 ; Zhou et al. 2019 ) , identify performance anomalies ( Doray and Dagenais 2017 ; Xu et al. 2021 ; Liu et al. 2020 ) , and diagnose root causes ( Murali et al. 2021 ; Li et al. 2021 ; Yu et al. 2021 ) . Therefore, many industry leaders have embraced various tracing solutions, including OpenTelemetry ( Opentelemetry 2025 ) , Skywalking ( Skywalking 2025 ) , and Jaeger ( Jaeger 2025 ) , demonstrating the technology’s widespread adoption ( Guo et al. 2020 ; Cai et al. 2019 ; Zhang et al. 2022c ) . Although distributed traces offer valuable information for system analysis, their extensive volume and the resulting storage requirements create significant obstacles. Alibaba’s e-commerce platform generates between 18.6 and 20.5 pebibytes (PB) of trace data each day ( Huang et al. 2025 ) . This immense amount arises because developers seek to capture the full spectrum of application behaviors to facilitate the diagnosis of emerging issues. However, persistently storing such a large amount of trace data incurs substantial operational overhead. To address these challenges, trace-level sampling techniques ( Gias et al. 2023 ; Zhang et al. 2023 ; Las-Casas et al. 2019 ; Huang et al. 2021 ; Huang et al. 2024 ) have been devised to selectively retain traces relevant to anomalous behavior ( Yu et al. 2021 ; Doray and Dagenais 2017 ; Liu et al. 2020 ) . Figure 1 . Comparisons between trace- and span-level sampling. In this paper, we designate the trace-level sampling as “trace sampling 1.0”. As shown in Fig. 1 -(a), the primary mechanism of trace-level sampling involves initially identifying the traces to be sampled and subsequently retaining only those selected traces while completely eliminating the remainder, a process we call the “1 or 0” strategy. These techniques are generally classified into head sampling ( Kaldor et al. 2017 ; Sigelman et al. 2010 ) or tail sampling ( Las-Casas et al. 2019 ; Huang et al. 2021 ; Huang et al. 2024 ) , depending on when the sampling decision occurs and the criteria applied. However, this approach of wholly discarding unsampled traces reveals significant shortcomings in practical scenarios. Existing research ( Huang et al. 2025 ) indicates that SREs may still need to access these discarded traces, as the traits of traces that require analysis are often unpredictable, leading to a query miss rate of up to 27.17\% . Moreover, such trace query failures can substantially hinder diagnostic approaches based on comparing normal and abnormal traces ( Doray and Dagenais 2017 ; Yu et al. 2021 ; Liu et al. 2020 ) . To address the limitations of trace-level sampling and enhance sampling flexibility, we introduce the concept of span-level sampling, which we designate as “trace sampling 2.0”. The fundamental observation behind span-level sampling is that most spans within a trace are irrelevant to explaining the performance variations under scrutiny ( Kaldor et al. 2017 ; Zhou et al. 2021 ; Ding et al. 2015 ) . For example, a study of Alibaba’s trace data indicates that 90% of spans contribute little meaningful information, while only 10% are critical for diagnosing performance issues ( Luo et al. 2021 ) . This suggests the potential to balance trace cost and utility at the span level by preserving spans that aid fault diagnosis while discarding those that do not. However, achieving a balance between trace cost and utility at the span level is far from straightforward. Transitioning from the trace-level sampling approach, an intuitive strategy might involve analyzing variations in span duration and retaining spans that significantly deviate from typical latency patterns. Nevertheless, we observed that relying solely on duration data overlooks critical invocation details within the trace, such as the specific services or functions traversed. For example, as illustrated in Figure 4 -(a) and (b), Trace-1 and Trace-2 represent different request types, but after sampling, their trace structure becomes identical, which makes their request types indistinguishable, thus affecting downstream diagnostic tasks (§ 3 ). To overcome this limitation, we propose Autoscope, the first span-level trace sampling method designed to advance Trace Sampling 2.0. The core idea of this work lies in precisely extracting execution logic from the intricate source code of a distributed system to enhance the span-level sampling process. Specifically, Autoscope begins by constructing a Call Site Control Flow Graph (CSCFG) through static analysis. Since static analysis alone struggles to capture cross-service calls in microservices architectures, Autoscope integrates runtime data for optimization. During the sampling phase, Autoscope maps span to their corresponding code functions and partitions trace data based on the CSCFG. This enables the identification of Dominant Span Sets (DSS), which leverages inference relationships between spans. By recording just one span within a DSS, the entire set can be inferred, significantly improving sampling efficiency. To ensure the representativeness of sampled spans, Autoscope employs a robust Z-score anomaly detection method to quantify span anomalies and select spans at the DSS level. Additionally, Autoscope adopts an incremental path-matching strategy to optimize execution path matching, further enhancing the completeness and accuracy of sampled trace data. We evaluated the performance of Autoscope on two open-source microservice applications. The results

Read the complete text