Energy Consumption of Dataframe Libraries for End-to-End Deep Learning Pipelines – A Comparative Analysis Conference: 10th International Workshop on Green and Sustainable Software; April 14,2026; Rio de Janeiro, Brazil 10th International Workshop on Green and Sustainable Software DOI: XXXXXXX.XXXXXX
Punit Kumar, Asif Imran, Tevfik Kosar · 2025
Read Energy Consumption of Dataframe Libraries for End-to-End Deep Learning Pipelines – A Comparative Analysis Conference: 10th International Workshop on Green and Sustainable Software; April 14,2026; Rio de Janeiro, Brazil 10th International Workshop on Green and Sustainable Software DOI: XXXXXXX.XXXXXX by Punit Kumar, Asif Imran, Tevfik Kosar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08644.txt.
Introduction
Punit Kumar, Asif Imran, and Tevfik Kosar measure what dataframe libraries cost inside full deep learning pipelines, not just in isolated benchmarks. They compare Pandas, Polars, and Dask, the three big Python tools for tabular data, while models train and run inference on real workloads.
The measurements cover runtime, memory use, disk use, and energy drawn by both CPU and GPU during loading, preprocessing, and batch feeding. These stages are easy to overlook, yet they sit on the critical path of every training run and can starve expensive graphics hardware. By embedding the libraries in complete pipelines across several models and datasets, the study shows where each tool helps or hurts total efficiency.
Catalog record
| CREATOR | Punit Kumar, Asif Imran, Tevfik Kosar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Energy Consumption of Dataframe Libraries for End-to-End Deep Learning Pipelines – A Comparative Analysis Conference: 10th International Workshop on Green and Sustainable Software; April 14,2026; Rio de Janeiro, Brazil 10th International Workshop on Green and Sustainable Software DOI: XXXXXXX.XXXXXXX Punit Kumar, Asif Imran, Tevfik Kosar Abstract . This paper presents a comparative performance analysis of three popular Python data manipulation libraries—Pandas, Polars, and Dask—within the context of deep learning training pipelines. The existing studies in this area do not embed the libraries inside a full deep-learning training pipeline where data loading , preprocessing , and batch feeding interact tightly with GPU workloads. To bridge this gap, we integrate Pandas, Polars, and Dask into representative deep learning training and inference pipelines and conduct experiments across a wide range of various machine learning models and datasets, measuring key performance indicators such as runtime, memory usage, disk usage, and energy consumption (CPU and GPU). Our comprehensive analysis reveals that Polars consistently minimizes CPU energy consumption on larger workloads, while Pandas remains competitive for moderate sizes. Dask’s overhead can lead to higher energy usage on small to moderate datasets. All three libraries achieve similar runtimes for heavy GPU workloads (ResNet, Mask R-CNN). Polars and Pandas maintain lower CPU memory footprints than Dask, but Dask offers easier scalability if data truly exceeds available RAM. Polars shows marginal energy savings on the CPU during preprocessing. 1. Introduction Efficient data handling is crucial in the development and deployment of deep learning models. Modern deep learning workflows often involve large-scale data preprocessing, feature engineering, and iterative model training. In Python, several libraries exist to facilitate data manipulation, each offering different trade-offs between ease of use, memory footprint, parallelism, and scalability. Pandas ( Gupta and Bagchi 2024 ) is a widely adopted library known for its intuitive API and rich functionality for structured data. However, it can become a bottleneck when datasets exceed main memory or when operations are not vectorized. Polars ( Lutkebohmert et al. 2021 ) , a more recent entrant, leverages Apache Arrow under the hood and employs parallel execution by default, promising significant performance improvements over Pandas for both in-memory and out-of-core tasks. Dask ( Rocklin 2015 ) , on the other hand, provides a framework for parallel and distributed computing by scaling Pandas operations to multiple cores or even clusters, making it suitable for larger-than-memory datasets. Several groups benchmark DataFrame backends, but mostly in isolation or with simple analytics workloads. Mozzillo et al. evaluate seven libraries on four real-world tasks ( Mozzillo et al. 2023a ) ; Broihier et al. propose PandasBench , a corpus of 102 notebooks for reproducible testing ( Broihier et al. 2025 ) ; and Bergamaschi et al. study energy-to-solution for Polars and Pandas ( Bergamaschi et al. 2024 ) . None of these works embeds the libraries inside a full deep-learning training pipeline where data loading , preprocessing , and batch feeding interact tightly with GPU workloads. To bridge this gap, we integrate Pandas, Polars, and Dask into representative pipelines for Random Forest , SVR , TabNet , XGBoost , ResNet-50 , Mask R-CNN , and Neural Collaborative Filtering (NCF) across datasets ranging from an insurance policy table to COCO images . We record runtime, peak CPU/GPU memory, disk I/O, and energy. CPU events are captured with Linux perf ( Linux Kernel Community 2024 ) ; GPU power is sampled via NVIDIA pynvml ( NVIDIA Corporation 2024 ) . Each configuration runs five times, and we report the trimmed mean and 95% confidence intervals to suppress outliers. Figure 1. Flowchart to highlight the experimental process of this study. This paper makes novel contributions to the field by answering the following research questions: • RQ1: How do Pandas, Polars, and Dask differ in performance, scalability, and energy efficiency when applied across full machine learning workflows? We present a complete comparison framework of Dataloader (Pandas, Polars, and Dask) within machine learning workflows, ranging from classical models to deep neural networks. Unlike prior work that focused on isolated steps (for example, data loading or training only), we evaluate their impact across the entire training and inference pipeline. • RQ2: How do modern dataframe libraries truly impact machine learning performance? While existing studies have examined Pandas, Polars, and Dask in isolation—often focusing solely on data loading or preprocessing—a critical gap remains: their comprehensive influence across complete ML workflows. We address this by conducting the first end-to-end comparison spanning classical models to deep neural networks, measuring performance from initial data ingestion through final inference. • RQ3: How do different dataloader configurations impact the computational energy efficiency? We go beyond runtime analysis and include detailed measurements of CPU and GPU energy consumption, using standardized tools like perf stat and pynvml ( NVIDIA Corporation 2024 ) . Our framework also tracks memory usage (RAM and VRAM), providing a complete energy profile for each dataloader setup. We evaluate these dataloaders on a diverse set of datasets, from small (Insurance ( Veeramani 2017 ) , ML-1M) to large-scale COCO images and annotations (from the Microsoft COCO ( Lin et al. 2014 ) ) and offer practical recommendations on when to use each backend based on dataset size, model type, and hardware configuration. This framework quantifies energy efficiency across data loading frameworks in the context of full ML pipelines, helping bridge the gap between performance benchmarking and sustainable machine learning. The rest of the paper is organized as follows: Section 2 explains the experiments, which include a comparative analysis of the dataloaders and a breakdown of the energy profiling. Section 3 highlights the integration of backends into the training pipelines and describes performance metrics. Section 4 presents the results of the experiments and summarizes our findings. Section 5 discusses the related work, and Section 6 concludes the paper. 2. Experimental Setup 2.1. Hardware and Software Environment All experiments are conducted on a workstation equipped with a 16-core, 32-thread CPU clocked at 2.10 GHz, 128 GB of RAM, an NVIDIA Tesla V100 GPU with 32 GB HBM2 memory, and a 1 TB NVMe SSD, running Ubuntu 20.04 LTS with Linux kernel 5.4. The software environment is managed using Conda and includes the following key libraries and tools: Python 3.8.12, Pandas 1.4.2, Polars 0.10.18, Dask 2022.2.0, TensorFlow 2.8.0, PyTorch 1.12.1, scikit-learn 1.0.2, XGBoost 1.5.0, PyTorch-TabNet 3.1.1, the Linux perf tool (version 5.4), and the NVIDIA Management Library Python bindings (pynvml) version 11.5.50. Energy and power measurements during experiments can be affected by various external factors. To minimize such influence and promote consistency, each experiment is executed five times, and the average result is used, following the methodology described by Georgiou et al. ( Georgiou et al. 2022 ) . As shown in Figure 1 , after completing each task on the dataset, the system is allowed to idle for 40 seconds using the "sleep" command. This pause aligns with the recommendation by Bornholt et al. ( Bornholt et al. 2012 ) , who note that allowing the system to idle for at least 30 seconds helps eliminate residual power effects and enables the system to return to a stable baseline before initiating the next task. 2.2. Datasets We select three representative datasets described below. The reason for their choice mainly depends