Waiting at the front door: Continuous monitoring of latency in the host network stack CCS: Networks End nodes CCS: Networks Network monitoring CCS: Networks Network performance analysis

Simon Sundberg, Anna Brunstrom, Simone Ferlin-Reiter, Jesper Dangaard Brouer, Toke Høiland-Jørgensen · 2026

Read Waiting at the front door: Continuous monitoring of latency in the host network stack CCS: Networks End nodes CCS: Networks Network monitoring CCS: Networks Network performance analysis by Simon Sundberg, Anna Brunstrom, Simone Ferlin-Reiter, Jesper Dangaard Brouer, Toke Høiland-Jørgensen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02057.txt.

Introduction

Sundberg and co-authors build a tool to watch latency inside host machines. Networks now chase sub-millisecond response times. At that scale, the end host itself adds delay. The link between card and CPU can be slow. The kernel network stack adds its own delays. Scheduling of applications adds still more delay.

Past work studied host latency at scattered levels. No tool watched it continuously in production use. Their netstacklat tool fills that gap directly. It captures latency at several points in the host. Points run from early kernel stages to application reads. Tests span 144 variants of HTTP workloads. Both Nginx and Apache servers are covered. The tool shows where host delay truly arises.

Catalog record

CREATORSimon Sundberg, Anna Brunstrom, Simone Ferlin-Reiter, Jesper Dangaard Brouer, Toke Høiland-Jørgensen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.02057

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Waiting at the front door: Continuous monitoring of latency in the host network stack CCS: Networks End nodes CCS: Networks Network monitoring CCS: Networks Network performance analysis Simon Sundberg, Anna Brunstrom, Simone Ferlin-Reiter, Jesper Dangaard Brouer, Toke Høiland-Jørgensen Abstract . With networking moving into the sub-millisecond latency domain, latency in the end host itself can become a significant barrier to achieving consistently low application latency. Both the physical interconnect between the network card and the CPU, the kernel network stack, and the scheduling of applications themselves can be considerable sources of latency. Previous work has studied host latency at various levels, yet there remains a lack of methods and tools to continuously monitor host latency in production. To remedy this, we present netstacklat, a monitoring tool that captures latency at several points in the host network, from the early parts of the Linux kernel network stack all the way until the application reads the data. We evaluate netstacklat in a testbed, demonstrating its ability to capture host latency across 144 variations of HTTP workloads for Nginx and Apache, while also showing how the low monitoring overhead does not inflate tail latency by more than 6%, where previous monitoring solutions increase it by over 100%. Furthermore, we share our initial findings from deploying netstacklat in Cloudflare’s global CDN network. † † cc-license: by 1. Introduction Traditionally, end-to-end latency has been attributed mainly to network congestion, where queueing occurs primarily in the network fabric. However, as network speeds continue to outpace host I/O and memory bandwidth improvements, hosts themselves have become critical points of contention. Many studies have shown that the host can have a significant impact on the end-to-end latency, for example due to congestion in the interconnect between the Network Interface Card (NIC) and CPU ( 1 ; 2 ; 41 ) , architectural inefficiencies in the Linux network stack ( 23 ; 9 ; 10 ; 3 ; 22 ) , interrupt processing ( 5 ; 15 ; 8 ) , and container and virtualization overhead ( 14 ; 28 ; 30 ; 11 ) . As such, the host network latency, which encompasses the delay from a packet arriving at the NIC, being transferred to the CPU, traversing the kernel’s network stack, and finally being read by the application, is becoming increasingly important as modern networks push for sub-millisecond performance. Being able to continuously monitor the host network latency would therefore be highly useful to identify local latency issues that could have a large impact on the end-to-end performance. Yet, despite this wealth of work examining the impact the host has on network latency, there remains a lack of methods to continuously monitor host network latency. While actively measuring end-to-end application latency ( 14 ; 5 ; 28 ; 10 ; 30 ; 8 ; 3 ; 22 ) , analyzing packet captures from external tap devices ( 15 ; 11 ) , or patching the kernel to log packet timestamps ( 23 ; 9 ) are suitable methods to infer host latency in controlled experiments, they are of little use for continuously monitoring the host network latency on servers running in production. To address this gap, we therefore make the following contributions: (1) We design and implement netstacklat, a tool able to monitor the latency buildup as packets traverse the ingress network stack, from early parts in the kernel’s receive path all the way up to application reception. By combining the Linux kernel’s timestamping mechanism with eBPF, netstacklat is able to track host latency for every packet system-wide with a low enough footprint to enable continuous monitoring. (2) We evaluate netstacklat in a testbed across 144 Nginx and Apache workloads, showing how netstacklat can track long tail latencies at several levels in the kernel network stack. Furthermore, we demonstrate that netstacklat has reasonably low overhead, only averaging 0.81% CPU utilization across all workloads, and only inflating end-to-end tail latency by 6% in a heavily loaded scenario where other state-of-the-art solutions increase tail latency by over 100%. (3) We present early insights from deploying netstacklat at Cloudflare, illustrating how host network latency increases as a server experiences progressively higher load. We also show how netstacklat uncovered an anomalous event where the host latency continuously increased over a 3-hour period before returning to normal. The rest of the paper is structured as follows: Section 2 covers existing work on monitoring host network latency. Section 3 provides a brief background on the Linux ingress network stack and its support for packet timestamping. Section 4 describes the design of netstacklat. In Section 5 , we evaluate netstacklat on a testbed with several HTTP workloads and show how the network stack latency and the monitoring overhead vary across all configurations. Section 6 compares the monitoring overhead of netstacklat with other tools that can be used for inferring network stack latency. In Section 7 , we share some initial insights from the deployment of netstacklat at Cloudflare. Section 8 discusses current limitations with netstacklat and how future work can address them. Finally, Section 9 concludes the work. 2. Related work While many works examine the impact of host network latency on the end-to-end performance, as covered in the introduction, the body of work focusing on monitoring aspects of host network latency is comparatively smaller. The most comprehensive monitoring solution in this vein is likely Google’s Fathom ( 31 ) . Fathom expands on Google’s earlier work with Dapper ( 35 ) for distributed monitoring of Remote Procedure Calls (RPCs) by collecting timestamps at several places in the network stack and application layer on both the client and server side. These timestamp-enriched traces provide a detailed breakdown of latency across the entire RPC chain. Furthermore, Fathom also offers automated root cause analysis for performance issues using a waterfall heuristic, and characterization of applications’ typical network performance through a Gaussian Mixture Model (GMM). While Fathom offers extensive monitoring capabilities, it is limited to monitoring RPC calls using one of their instrumented RPC libraries in scenarios where both the client and server can be monitored. Furthermore, for the highly granular data collection to scale for fleet-wide monitoring, Fathom uses aggressive sampling, monitoring only 1:1000 or 1:128,000 RPC calls. A more generic tool for monitoring latency in the host network is NSight ( 18 ) , which utilizes the Intel-PT ( 21 ) hardware profiler and NIC hardware timestamps to construct highly accurate per-message lifetimes with virtually no overhead. Unfortunately, at the time of writing, the NSight tool does not appear to be publicly available. While netstacklat cannot match neither the granularity of measurements nor the incredibly low overhead of Nsight, netstacklat offers an entirely software-based alternative that can be deployed without any kernel modifications. Furthermore, NSight is not well-suited for continuous monitoring due to limitations with the buffering in Intel-PT, and also produces upwards of 1 GB/s of monitoring data for a single host. A closely related work to netstacklat is lattrace ( 25 ) , which has been a strong inspiration during netstacklat’s development. Lattrace uses eBPF to record the time when SKBs reach many of the same points in the network stack as netstacklat. However, instead of aggregating latencies in-kernel, lattrace pushes timestamp records

Read the complete text