Single-Thread JPEG Decoder Benchmarks Mis-Evaluate ML Data Loaders
Vladimir Iglovikov, Dmitry Kosarevsky · 2026
Read Single-Thread JPEG Decoder Benchmarks Mis-Evaluate ML Data Loaders by Vladimir Iglovikov, Dmitry Kosarevsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08731.txt.
Introduction
Vladimir Iglovikov and Dmitry Kosarevsky benchmark JPEG decoders inside the data loaders that feed image model training, noting that the evidence title frames the same study around single thread claims. Fixing a PyTorch loader at eight workers, they compare twelve Python decoders against the standard Pillow library on four cloud processors and two photo workloads, repeating each setting five times in isolation.
On large camera originals, every decoder lands near Pillow speed. On a mixed workload with social media processed copies, three decoders run about forty percent faster, though the exact leader changes by processor. Practitioners learn to benchmark on photos like their own, since workload, not brand, decides the winner.
Catalog record
| CREATOR | Vladimir Iglovikov, Dmitry Kosarevsky |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08731
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Single-Thread JPEG Decoder Benchmarks Mis-Evaluate ML Data Loaders Vladimir Iglovikov, Dmitry Kosarevsky Abstract JPEG decode is routine ML infrastructure, but Python decoder choices are often justified by single-process, single-thread microbenchmarks. We audit this evaluation assumption with thirteen Python-accessible JPEG decode paths on five matched 16 vCPU Google Cloud CPUs: Intel Emerald Rapids, AMD Zen 4, AMD Zen 5, ARM Neoverse V2, and ARM Neoverse N1. ImageNet validation is the workload, not a new dataset contribution: each run decodes the full 50,000-image split from memory and reports single-thread throughput for all decoders, PyTorch DataLoader throughput for eligible decoders at worker counts { 0 , 2 , 4 , 8 } \{0,2,4,8\} , and decoder skip behavior. The evaluation protocol changes the supported conclusion. On Neoverse V2, imageio is ninth in single-thread throughput yet lands in the top DataLoader tier with torchvision ; on Zen 4, torchvision rises from seventh single-thread to the top measured DataLoader tier; on Neoverse N1, imagecodecs is the single-thread leader but fifth at peak DataLoader throughput. We also find that worker-count conclusions differ between Zen 4 and Zen 5, TensorFlow has a large single-thread ARM penalty, and strict native JPEG decoders/wrappers reject the same rare ImageNet JPEG. For PyTorch DataLoader workloads, torchvision and simplejpeg form the strongest measured zero-skip tier: torchvision has the highest mean normalized throughput, while simplejpeg has the highest minimum. OpenCV remains a robust general-purpose fallback above 90% of the platform-local winner on every tested CPU. We release raw JSON, generated tables/figures, and an executable local/cloud benchmark framework. 1 Introduction Image loading becomes visible when it caps training throughput. When decode cannot feed the learner, GPUs idle, queue depth collapses, and adding model-side compute stops helping. A natural response is to benchmark decoders in a tight loop, choose the fastest images/s number, and move on. That workflow is attractive because single-thread microbenchmarks are cheap, deterministic, and easy to compare. The problem is that the evaluation design encodes a claim: single-thread decoder throughput identifies the right decoder for training data loaders. That claim is often false. Training pipelines usually pay JPEG decode cost inside a DataLoader , with multiple worker processes, batches, Python/C boundary effects, library thread pools, fork behavior, and CPU-specific scheduling. A benchmark that collapses those conditions into one isolated decoder call can support the wrong library choice or hide a robustness policy that matters operationally. This paper treats the decoder benchmark itself as the object of evaluation. Evaluation practice in ML includes the processes, tools, datasets, benchmarks, and assumptions used to test, compare, and interpret systems. Our contribution is in that category: we compare a common evaluation shortcut against a loader-aware protocol, show where the shortcut fails under loader parallelism, CPU generation, and robustness policy, and ship a runnable framework for evaluating the claim locally and on cloud CPUs. Contributions. 1. We compare the common single-thread decoder protocol against a PyTorch DataLoader protocol on the same workload, libraries, machines, and pinned environment. 2. We release the code as an installable CLI for local datasets and as Google Cloud runners for matched 16 vCPU replication. 3. We isolate rank changes and worker-scaling effects that disappear if evaluation uses only one CPU family or only one worker count. 4. We convert benchmark cells into an operational decision protocol that accounts for decoder, worker count, microarchitecture, and robustness policy. 2 Related work Input pipelines are a known bottleneck in large-scale training. Systems such as FFCV [ 4 ] , WebDataset [ 5 ] , TorchData [ 11 ] , NVIDIA DALI [ 6 ] , and nvJPEG [ 7 ] improve storage layout, sharding, CPU/GPU preprocessing, or GPU decode. These systems address broader pipeline design. Our study isolates a narrower but common primitive: Python-visible JPEG decoding from ordinary JPEG files inside a PyTorch-style CPU data loader. The decoder ecosystem itself is fragmented. Production code may use Pillow [ 9 ] , OpenCV [ 8 ] , torchvision [ 10 , 12 ] , TensorFlow [ 1 ] , Kornia [ 13 ] , wrappers around libjpeg-turbo [ 15 ] , or JPEGli-backed wrappers such as ajpegli [ 3 ] . Existing comparisons often emphasize single-machine single-thread throughput. That is useful for understanding a decoder implementation, but it does not define a complete evaluation for training-time library choice. 3 Evaluation design Workload. We use the full ImageNet validation split [ 14 ] : 50,000 JPEG files. ImageNet-val supplies a widely recognized JPEG size and color-distribution workload; the contribution is not a new dataset. The benchmark runs in memory mode so headline numbers isolate decode and loader overhead rather than disk, network, or object-store latency. Decoders and protocols. We evaluate thirteen Python-accessible decode paths: simplejpeg , turbojpeg , jpeg4py , kornia-rs , ajpegli , opencv , imagecodecs , pyvips , pillow , skimage , imageio , torchvision , and tensorflow . The harness normalizes successful outputs to RGB uint8 . We report single-thread memory decode, PyTorch DataLoader throughput for eligible decoders at worker counts \{0,2,4,8\} , and skip accounting. Hardware. We use five matched Google Cloud -standard-16 machine types [ 2 ] , selected to separate vendor, ISA, and generation effects: Intel Emerald Rapids, AMD Zen 4, AMD Zen 5, ARM Neoverse V2 / Google Axion, and ARM Neoverse N1 / Ampere Altra. Statistical policy. Tables report descriptive mean \pm sample standard deviation throughput. Comparative claims use the raw per-run throughput samples stored in JSON rather than only table means. We apply practical-significance thresholds before using strict faster/slower language: 1% for single-thread decode and 5% for DataLoader throughput, where worker-process variance is larger. When gaps are small or uncertain, we describe decoders as tied, top-tier, or not clearly separated. In Figure 4 , whiskers are platform min–max ranges, not statistical confidence intervals. Table 1: Protocol-selection guide: each deployment question requires matching evidence before it supports an operational claim. Deployment question Insufficient evaluation Required evaluation Supported claim Which decoder is fastest? Unqualified fastest claim Single-thread table with CPU/workload scope Component speed only Which decoder should feed PyTorch? Single-thread ranking DataLoader throughput Loader-scale top tier How many workers? One CPU family Worker sweep per CPU CPU-generation-specific worker policy Is it safe by default? Throughput only Skip/failure accounting Operational tier Table 1 summarizes the evaluation principle used throughout the paper: the same raw decoder measurements answer different deployment questions only when paired with a matching protocol. The benchmark is designed to make those claim boundaries explicit instead of reporting a single leaderboard. 4 Results 4.1 Single-thread protocol supports only component speed Single-thread measurements produce the expected component-speed picture: libjpeg-turbo-family decoders and OpenCV are usually near the top. By mean throughput, simplejpeg leads on Intel, Zen 4, and Neoverse V2; torchvision narrowly leads on Zen 5; imagecodecs narrowly leads on Neoverse N1. These single-thread leaders are descriptive, not universal recommendations. 4.2 DataLoader protocol changes the decoder conclusion When the same decoders run through DataLoader , the supported conclusion changes. The key finding is not that every top cell is statistically isolated. It is that the single-thread protocol fails to identify the loader-relevant top tier on several platforms. Figure 1: Protocol changes decoder recommendations. Bars show rank change from single-thread memory decode to peak PyTorch DataLoader throughput on representative Intel, AMD, and