Recommending Pre-Trained Models for IoT Devices
Parth V. Patil, Wenxin Jiang, Huiyun Peng, Daniel Lugo, Kelechi G. Kalu, Josh LeBlanc, Lawrence Smith, Hyeonwoo Heo, Nathanael Aou, James C. Davis · 2024
Read Recommending Pre-Trained Models for IoT Devices by Parth V. Patil, Wenxin Jiang, Huiyun Peng, Daniel Lugo, Kelechi G. Kalu, Josh LeBlanc, Lawrence Smith, Hyeonwoo Heo, Nathanael Aou, James C. Davis (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18972.txt.
Introduction
Parth Patil and co-authors recommend pre-trained models for internet devices in a 2024 paper. Pre-trained models speed machine learning deployment everywhere. Tricks like quantization extend them to small hardware. Choosing among many candidate models is itself costly, though.
Existing selection tools estimate task fit without full tuning. They largely ignore hardware limits in their rankings. That omission bites hardest on constrained devices. The paper catalogs where current recommenders fall short. A hardware-aware selection method is introduced in response. Memory, compute, and energy budgets enter the recommendation. The authors also propose a research agenda for the area. Effective systems must co-optimize fit and feasibility. Engineers drowning in model zoos are the audience. Future benchmarks should grade hardware realism. The work is part position paper, part method launch.
Catalog record
| CREATOR | Parth V. Patil, Wenxin Jiang, Huiyun Peng, Daniel Lugo, Kelechi G. Kalu, Josh LeBlanc, Lawrence Smith, Hyeonwoo Heo, Nathanael Aou, James C. Davis |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18972
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Recommending Pre-Trained Models for IoT Devices Parth V. Patil, Wenxin Jiang, Huiyun Peng, Daniel Lugo, Kelechi G. Kalu, Josh LeBlanc, Lawrence Smith, Hyeonwoo Heo, Nathanael Aou, James C. Davis Abstract The availability of pre-trained models (PTMs) has enabled faster deployment of machine learning across applications by reducing the need for extensive training. Techniques like quantization and distillation have further expanded PTM applicability to resource-constrained IoT hardware. Given the many PTM options for any given task, engineers often find it too costly to evaluate each model’s suitability. Approaches such as LogME, LEEP, and ModelSpider help streamline model selection by estimating task relevance without exhaustive tuning. However, these methods largely leave hardware constraints as future work—a significant limitation in IoT settings. In this paper, we identify the limitations of current model recommendation approaches regarding hardware constraints and introduce a novel, hardware-aware method for PTM selection. We also propose a research agenda to guide the development of effective, hardware-conscious model recommendation systems for IoT applications. I Introduction Many IoT applications today use deep neural networks (DNNs) for advanced functionality. Examples include object detection in autonomous vehicles [ 1 ] , crop health monitoring in agriculture [ 2 ] , and natural language processing for smart home assistants [ 3 ] . Given the cost of developing and training a DNN from scratch [ 4 ] , engineers often leverage pre-trained models (PTMs) to expedite development and deployment processes. However, as highlighted by recent studies [ 5 ] , selecting an appropriate PTM frequently requires manual evaluation of model performance on downstream tasks, a process that is time-intensive and prone to variability. Furthermore, a separate study [ 6 ] found that practitioners often encounter hardware constraints or lack the expertise needed to adapt DNN as the main hurdle. This underscores the value of a systematic approach to PTM recommendation, particularly one that addresses the resource limitations of IoT hardware. PTM selection extends beyond IoT, with prior work making significant strides in recommending PTMs for specific tasks without model fine-tuning. Figure 1 summarizes the current PTM selection pipeline. These methods fall into two categories: heuristic-based [ 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] and learning-based [ 15 , 16 , 17 ] , as detailed in II . However, their primary focus is maximizing model performance ( e.g., accuracy), with limited attention to hardware constraints. Deploying PTMs on IoT devices introduces many such constraints, including limited CPU, memory, energy, and low-bandwidth communication. Adapting current recommendation approaches to address these IoT-specific limitations is nontrivial, as many of these methods rely on measures such as class similarity between source and target tasks or the model’s capacity for distinguishing classes. Therefore, an effective IoT solution must incorporate both task suitability and hardware constraints, with potential further insights from modeling device energy consumption. We identify two critical gaps in the state-of-the-art PTM recommendation systems: (1) Lack of IoT-specific inputs for model recommendation (2) Lack of Ground-Truth rankings for IoT devices . Developing a hardware-aware recommender requires first establishing a ground truth model ranking on devices, which is essential for addressing the first gap. Fig. 1: A model recommendation process for selecting the best pre-trained model (PTM) for a target task. Notably, hardware specifications are not considered, limiting use on constrained devices. Models trained on a source dataset (X, Y) are stored in a model hub. For a target task with a non-overlapping dataset (X, Y), the system recommends the most suitable PTM for fine-tuning, resulting in the best model for the task. In this work, we introduce approaches and research agendas to address these issues. First, we propose two modifications to the Model Spider framework [ 15 ] to enable hardware-aware recommendations, which we term Model Spider Fusion and Model Spider Shadow . Both approaches aim to address the first gap, each with distinct methods. Additionally, we outline methods for collecting raw data by defining essential metrics and generating custom rankings based on these hardware-specific performance indicators. We close by discussing future opportunities to support the ongoing development of hardware-conscious model recommendation systems. Our contributions are: 1. We identify key gaps in PTM recommendation for IoT. 2. We introduce methods to address these gaps, focusing on hardware-specific metrics and tweaks to existing methods. 3. We outline a research agenda aimed at advancing PTM recommendations with a focus on hardware and sustainability. II background and related work This section covers background and related works on pre-trained model recommendation ( section II-A ) and model benchmarking in IoT systems ( section II-B ). II-A Works for Model Recommendation Researchers have developed various methods to help software engineers identify the most suitable PTMs, aiming to minimize fine-tuning and forward passes to reduce computational costs, as summarized in table I . This section reviews state-of-the-art approaches for PTM recommendations in downstream tasks, categorizing them into two main types: heuristic-based and learning-based . Heuristic-Based Methods : These methods typically devise a novel heuristic or scoring system to rank a PTM’s effectiveness for a target task. This category includes methods, such as NCE [ 7 ] , and H-Score [ 8 ] , which estimate the similarity between source and target labels directly. Additional methods, such as LEEP [ 9 ] , N-LEEP [ 10 ] , LogME [ 11 ] , PACTran [ 12 ] , GBC [ 13 ] , and LFC [ 14 ] , rely on a forward pass on the target dataset to capture representations. These representations are then analyzed for their usefulness based on target labels. However, heuristic approaches often struggle to capture the nuances of hardware specifications and their correlation with model performance, overlooking the practical challenges software engineers face when deploying models on hardware-constrained environments. For example, on IoT devices, larger models can leverage hardware-accelerated activation functions to boost performance, while smaller models with complex functions may still encounter bottlenecks. Learning-Based Methods : Recent methods, including Model Spider [ 15 ] , EMMS [ 16 ] , and Fennec [ 17 ] , employ machine learning to eliminate the need for forward passes. These methods encode both the PTM and target dataset into a latent space, using learning techniques to discern patterns and predict performance. Given their promising results, we focus on learning-based techniques for our hardware-aware recommendation solutions, as this approach is well-suited for software engineers to encode complex hardware specifications, such as CPU type, architecture, memory size, and I/O speed. Model Spider tokenizes tasks and pre-trained models to generate recommendations that balance efficiency and accuracy [ 15 ] . The Model Spider approach encodes each pre-trained model into a token \theta_{m} using an extractor \Psi , which encapsulates essential characteristics such as architectures and parameters. Simultaneously, tasks are embedded as other tokens \mu(T) , reflecting dataset statistics and task characteristics. The key contribution of Model Spider is its use of a multi-head attention mechanism to assess the similarity sim(\theta_{m},\mu(T)) between model and task tokens. This similarity score serves as a reliable indicator of model performance, enabling a ranked selection of pre-trained models aligned with the task requirements. Optionally, it also allows for the application of a forward pass, leading to a refined token \theta^{*}_{m} that further