CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
Irene Wang, Mostafa Elhoushi, H. Ekin Sumbul, Samuel Hsia, Daniel Jiang, Newsha Ardalani, Divya Mahajan, Carole-Jean Wu, Bilge Acun · 2025
Read CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization by Irene Wang, Mostafa Elhoushi, H. Ekin Sumbul, Samuel Hsia, Daniel Jiang, Newsha Ardalani, Divya Mahajan, Carole-Jean Wu, Bilge Acun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01386.txt.
Introduction
Irene Wang and eight co-authors present CATransformers, a framework that designs neural networks and their computer chips together with carbon emissions as a goal. Their paper came out in 2025. Artificial intelligence carries a lifetime carbon cost: energy burned in training and use, plus the footprint of manufacturing the hardware.
Their framework counts both costs early, while the model and chip are still being designed. Searching that joint space reveals trade-offs that speed-only or energy-only design misses. Across a range of transformer models, the method cut total emissions by up to about a third while holding accuracy and speed. A case study on models that mix text and images shows the approach extends beyond language. The code is released openly.
Catalog record
| CREATOR | Irene Wang, Mostafa Elhoushi, H. Ekin Sumbul, Samuel Hsia, Daniel Jiang, Newsha Ardalani, Divya Mahajan, Carole-Jean Wu, Bilge Acun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01386
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization Irene Wang, Mostafa Elhoushi, H. Ekin Sumbul, Samuel Hsia, Daniel Jiang, Newsha Ardalani, Divya Mahajan, Carole-Jean Wu, Bilge Acun Abstract Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. This growing adoption is expected to increase the associated lifecycle carbon footprint, including both operational carbon from training and inference and embodied carbon from AI hardware manufacturing. We introduce CATransformers —the first carbon-aware co-optimization framework for Transformer-based models and hardware accelerators. By integrating both operational and embodied carbon into early-stage design space exploration, CATransformers enables sustainability-driven model architecture and hardware accelerator co-design that reveals fundamentally different trade-offs than latency- or energy-centric approaches. Evaluated across a range of Transformer models, CATransformers consistently demonstrates the potential to reduce total carbon emissions—by up to 30%—while maintaining accuracy and latency. We further highlight its extensibility through a focused case study on multi-modal models. Our results emphasize the need for holistic optimization methods that prioritize carbon efficiency without compromising model capability and execution time performance. The source code of CATransformers is available at https://github.com/facebookresearch/CATransformers . 1 Introduction As machine learning (ML) systems become more widespread across various industries, it is crucial to take a closer examination of their carbon footprint and find strategies to mitigate it across the system stack. Sustainable ML system design requires a holistic approach that considers both operational carbon (energy used during training and inference) and embodied carbon (emissions from hardware manufacturing and lifecycle) Gupta et al. (2022a) , which together contribute to total emissions. This work tackles the question: How does incorporating carbon footprint metrics into optimization workflows influence the design of ML models and hardware architectures as a co-optimization? A fundamental challenge in designing sustainable AI systems stem from the tight coupling between model architecture and hardware design, which together shape both operational and embodied carbon. Model execution on a particular hardware determines runtime characteristics, such as compute intensity and memory access patterns, which affect operational carbon, while hardware parameters like chip area, memory hierarchy, and fabrication technology drive embodied carbon. Carbon optimization requires joint model-hardware exploration, particularly during early accelerator design, to uncover opportunities that align model demands with hardware capabilities in a carbon-efficient way. Co-design is even more important for multi-modal workloads like vision-language models. Each modality introduces distinct bottlenecks—e.g., vision transformers usually have compute-bound workloads Du et al. (2024) , while text transformers have more memory-bound workloads Fu (2024) . Co-optimization across modalities and system layers is essential to minimize the total carbon footprint without compromising performance. While this sustainability challenge applies across the entire AI lifecycle, the rapid adoption of AI on edge devices presents a particularly acute and fast-growing problem Qiu et al. (2023) ; Gupta et al. (2022b) . With billions of products like smartphones and AR/VR headsets requiring specialized, custom accelerators Wu et al. (2024) ; Sumbul et al. (2022) ; HAILO (2025) ; Google (2020) ; Nvidia (2025) , the embodied carbon from manufacturing constitutes a massive, upfront environmental cost. Furthermore, the cumulative operational carbon from continuous inference over a device’s multi-year lifespan represents a substantial and often overlooked energy demand Apple Inc. (2021) . Addressing the tightly coupled model-hardware system at the edge is therefore a critical first step, offering a high-impact domain to develop and validate the carbon-aware co-design principles needed for sustainable AI. Prior hardware-aware neural architecture search (NAS) methods have focused primarily on latency or energy and are not suitable for total carbon optimization. This is because carbon is a complex metric that cannot be captured by optimizing latency or energy in isolation. These approaches either neglect embodied carbon by optimizing only the model on fixed hardware Wang et al. (2020) or treat latency, energy, and chip area as orthogonal constraints Zhou et al. (2022) , overlooking the tightly coupled nature of operational (latency- and energy-driven) and embodied (chip area-driven) carbon. As a result, these methods are fundamentally limited in their ability to minimize total carbon emissions. Furthermore, prior work often assumes fixed hardware or fixed model backbones, limiting their applicability to a broader range of emerging hardware platforms and modern, multi-modal models. We introduce CATransformers — the first framework to jointly optimize model and hardware architecture with the specific goal of minimizing total carbon emissions for edge inference-only devices. Distinct from prior latency- or energy-centric methods, our approach incorporates both operational and embodied carbon into a unified optimization pipeline , revealing carbon-efficient configurations previously overlooked. We observe that optimizing traditional metrics in isolation frequently results in suboptimal trade-offs. For example, accelerating inference by increasing compute density may inadvertently increase power consumption and embodied carbon due to larger chip area or more resource-intensive fabrication. By prioritizing total carbon as a first-class objective, CATransformers yields fundamentally different and more sustainable design choices. CATransformers systematically explores the joint design space of model architectures and hardware accelerators using multi-objective optimization. It comprises three main components: (1) a Multi-Objective Bayesian Optimizer that balances accuracy, latency, energy, and carbon emissions; (2) an ML Model Evaluator that efficiently navigates model variants using importance-based pruning and fine-tuning; and (3) a Hardware Estimator that profiles latency, energy, and carbon footprint. We evaluate CATransformers on language, vision, and multi-modal Transformer-based models and show that its co-optimized model–hardware configurations reduce total carbon by 30% over latency-optimized and 8% over energy-optimized baselines. To further highlight its utility, we present CarbonCLIP, a family of CATransformers -optimized CLIP models. Figure 1 : CarbonCLIP models achieve lower carbon footprint and higher accuracy compared to baseline CLIP models. Figure 1 shows CarbonCLIP achieves up to 17% lower total carbon emissions compared to state-of-the-art CLIP variants on edge devices, while maintaining similar accuracy and latency. Joint optimization enables low-carbon, high-accuracy designs that outperform approaches that optimize only hardware configurations. Our key contributions include: 1. Insights and Analysis: Empirical insights showing how carbon-aware optimization shifts model–hardware trade-offs relative to traditional metrics, enabling early-stage design exploration for next-generation ML accelerators. 2. Quantification Framework : We develop the first open-source toolchain to estimate both operational and embodied carbon for custom accelerators during design-space exploration. 3. Carbon-Aware Co-optimization: CATransformers uses multi-objective Bayesian optimization to jointly explore model and hardware design spaces, balancing accuracy, latency, energy, and carbon. It leverages fine-tuning–based proxy signals to reduce evaluation cost. 4. Sustainable Multi-Modal Models : Using CATransformers , we co-optimize CLIP variants (CarbonCLIP) that reduce total carbon by up to 17% compared to edge-deployed CLIP baselines, without sacrificing accuracy or latency. CATransformers is designed as a modular, extensible framework that supports the seamless integration of diverse model architectures, optimization strategies, and carbon estimation techniques. The goal is not to outperform other NAS frameworks on traditional metrics, but to highlight the fundamentally different design trade-offs that emerge under carbon-centric optimization objectives. Through systematic comparisons across multiple optimization modes and model families, we demonstrate the practical impact of carbon-aware design, which is the core contribution of this work. 2 Background and Related Works Hardware Accelerator Search: Specialized hardware accelerators have been developed to efficiently run deep learning workloads, with tensor cores for matrix operations Jouppi et al. (2023) ; Mahajan et al. (2016) ; Park et al. (2017) ; Chen et al. (2016) and