Green MLOps: Closed-Loop, Energy-Aware Inference with NVIDIA Triton, FastAPI, and Bio-Inspired Thresholding ∗ PhD, Co-founder, InnoDeep

Mustapha HAMDI ∗, Mourad JABOU · 2026

Read Green MLOps: Closed-Loop, Energy-Aware Inference with NVIDIA Triton, FastAPI, and Bio-Inspired Thresholding ∗ PhD, Co-founder, InnoDeep by Mustapha HAMDI ∗, Mourad JABOU (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04250.txt.

Introduction

Mustapha Hamdi and Mourad Jabou propose serving AI models under a closed energy-aware loop inspired by protein folding. Running inference for long periods can outweigh training in total carbon cost, they note. Their framework maps folding energy basins onto inference cost landscapes and admits each request only when its expected gain justifies its energy cost.

A decaying threshold biases the system toward the first acceptable low-cost basin instead of chasing costly global optima. They test DistilBERT and a small ResNet served through FastAPI with ONNX Runtime and NVIDIA Triton on a stated graphics card. The controller cuts processing time by a large share against open-loop serving, with accuracy loss held under half a percent in their ablation. They also map where light local serving beats managed batching.

Catalog record

CREATORMustapha HAMDI ∗, Mourad JABOU
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04250

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Green MLOps: Closed-Loop, Energy-Aware Inference with NVIDIA Triton, FastAPI, and Bio-Inspired Thresholding ∗ PhD, Co-founder, InnoDeep Mustapha HAMDI ∗, Mourad JABOU Abstract Energy efficiency is a first-order concern in AI deployment, as long-running inference can exceed training in cumulative carbon impact. We propose a bio-inspired framework that maps protein-folding energy basins to inference cost landscapes and controls execution via a decaying, closed-loop threshold. A request is admitted only when the expected utility-to-energy trade-off is favorable (high confidence/utility at low marginal energy and congestion), biasing operation toward the first acceptable local basin rather than pursuing costly global minima. We evaluate DistilBERT and ResNet-18 served through FastAPI with ONNX Runtime and NVIDIA Triton on an RTX 4000 Ada GPU. Our ablation study reveals that the bio-controller reduces processing time by 42% compared to standard open-loop execution (0.50s vs 0.29s on A100 test set), with a minimal accuracy degradation ( < 0.5 % <0.5\% ). Furthermore, we establish the efficiency boundaries between lightweight local serving (ORT) and managed batching (Triton). The results connect biophysical energy models to Green MLOps and offer a practical, auditable basis for closed-loop energy-aware inference in production. I Introduction Production AI is not a single heroic training run; it is a never-ending stream of inference calls. In many applications, lifecycle energy is dominated by serving and data movement rather than training, elevating inference engineering into a sustainability problem. Prior analyses have called for energy transparency and methodological discipline in reporting consumption and emissions, while emphasizing system-level levers such as batching, model choice, and hardware placement. This work contributes an operational recipe: (1) a dual-path serving stack—FastAPI + ONNX Runtime (ORT) for low-latency local execution, and NVIDIA Triton for managed batching and multi-framework serving; (2) instrumentation via MLflow and CodeCarbon; and (3) a closed-loop controller that decides whether to execute or skip an inference based on a bio-inspired energy threshold that evolves with load, extending the theoretical bio-physical frameworks proposed in [ 7 ] . Triton’s dynamic batching and scheduler queues are leveraged for throughput under concurrency; ORT’s device-tensor and performance tuning options help minimize host–device penalties in the local path. We evaluate two canonical models: DistilBERT and ResNet-18. Fig. 1: Bio-inspired optimization over states/model confidence t . The cost J(x) landscape is gated by a decaying threshold \tau(t)=\tau_{\infty}+(\tau_{0}-\tau_{\infty})e^{-kt} ; the controller admits points in the local stable basin (shaded) and skips high-cost paths toward the global minimum. II Related Work Serving systems. Building on the concepts from Gopalan (2025) [ 5 ] toward comparative benchmarking of inference frameworks (Triton, TensorRT, ONNX Runtime, FastAPI), modern serving focuses on latency SLOs, throughput, and utilization. Early systems like Clipper introduced low-latency prediction serving with adaptive batching and model selection. NVIDIA Triton advances this with backends for ONNX, TensorRT, PyTorch, and graph-level scheduling including dynamic batching and instance groups, exposing HTTP/gRPC endpoints and model repositories. Energy-aware ML. Policy and measurement frameworks emphasize reporting energy and emissions, motivating tools such as CodeCarbon and lifecycle tracking in MLflow. We adopt CodeCarbon for kWh/CO 2 estimation and MLflow for experiment lineage and comparability. Bio-inspired control. Energy landscapes in protein folding illuminate how complex systems navigate toward acceptable local minima [ 3 ] . Recent works like StructuredDNA [ 7 ] apply these biophysics to Transformer routing. Similarly, SGEMAS [ 6 ] utilizes entropic homeostasis for anomaly detection. We unify these concepts to design a thresholded controller that filters low-utility inference work when the current operating point already lies in a satisfactory basin. III Models and Serving Architecture III-A Models DistilBERT: distilled from BERT, retaining much of its language understanding with fewer parameters and lower latency; sequence length 128. ResNet-18: residual connections facilitate training of deep CNNs; the 18-layer variant is a stable reference for image classification at 224 \times 224. III-B Dual-path serving Path A (FastAPI + ORT). A lightweight REST layer orchestrates local ORT inference. ORT’s performance guidance (I/O binding, execution providers, device tensors) reduces CPU–GPU copies. Ideal for small batches and low concurrency. Path B (NVIDIA Triton). Models are placed in a Triton model repository with config.pbtxt enabling max_batch_size and dynamic batching, allowing the scheduler to fuse requests into GPU-efficient batches under load. Instance groups can exploit multiple GPU streams. Excels for bursty or sustained higher QPS. III-C Instrumentation MLflow logs latency statistics, throughput, and controller state. CodeCarbon estimates energy (kWh) and CO 2 , with GPU power via NVML. Results are exported alongside MLflow metrics. IV Closed-Loop Thresholding: Formulation Fig. 2: Closed-loop, dual-path serving architecture (controller, FastAPI+ORT path, Triton path) with feedback via MLflow and CodeCarbon updating \tau(t) . For a given request x , define the cost functional J(x)=\alpha\,L(x)+\beta\,E(x)+\gamma\,C(x), (1) where L(x) is an uncertainty or loss proxy (e.g., entropy), E(x) is marginal energy, and C(x) reflects congestion/queue penalty (e.g., current batch fill level, recent tail latency). A request is admitted for inference iff J(x)\geq\tau(t), (2) with a time-varying threshold \tau(t) that decays from permissive to strict as the system stabilizes: \tau(t)=\tau_{\infty}+(\tau_{0}-\tau_{\infty})\,e^{-kt},\quad k>0. (3) At startup, tolerate more exploration (higher \tau ); once the system is in a basin with acceptable service/energy trade-offs, tighten admission to prune low-utility work, preventing wasteful oscillations. Notes on proxies. L(x) : softmax entropy or margin; E(x) : rolling average of joules per request; C(x) : function of queue depth, recent P95 latency, or Triton’s accumulated microbatches. IV-A The Closed-Loop Controller: Operation and Biological Analogy The controller is an active triage system whose objective is to improve energy efficiency by limiting demand rather than maximizing supply. The Principle: Avoiding Energy Waste. The controller mimics protein-folding energy landscapes: a protein reaches an acceptable local energy minimum (functional shape) without pursuing the absolute global minimum if the path is too costly or unstable. In MLOps, this translates to finding an energy-efficient, stable serving regime and rejecting requests that threaten this stability. TABLE I: Biology \leftrightarrow MLOps \leftrightarrow Controller behavior Biological Element MLOps Analogy Controller Behavior Energy Landscape The space of operational states, where the Y-axis is the Cost J(x) . The system seeks to navigate toward a low-cost basin (minimum) . Folding Stabilization of the inference system (adjusting batch sizes, queues). The threshold \tau(t) decreases to tighten admission once stability is reached. Local Energy Minimum Acceptable Operational State (e.g., low latency AND low kWh/request). The system only admits requests x for which J(x)\geq\tau(t) , ensuring they do not “push the total cost uphill. Costly Transitions Queue oscillations, scheduler thrashing, GPU context switching. The \tau(t) filter rejects requests with high C(x) (congestion penalty), protecting the stable state. Dynamic threshold \tau(t) . The threshold decays exponentially: \tau(t)=\tau_{\infty}+(\tau_{0}-\tau_{\infty})\,e^{-kt},\quad k>0 \tau_{0} (initial): high at t{=}0 , permissive to explore and reach a serving state. \tau_{\infty} (limit): lower after stabilization; only high-utility or very low-cost requests are admitted, reducing energy waste from marginal calls. The cost function J(x) . J(x) decides per request x : J(x)=\alpha\,L(x)+\beta\,E(x)+\gamma\,C(x) A) L(x) (utility/uncertainty). Role: value or risk of inference. Proxies: softmax entropy; 1{-}\text{confidence} . Rationale: for given \beta,\gamma , admit high-uncertainty (useful) requests; reject those already highly confident. B) E(x) (marginal energy). Role: joules/kWh to execute x . Proxy: rolling joules/request from CodeCarbon+NVML. Rationale: if E(x) spikes, only very valuable or very low-cost requests pass. C) C(x) (congestion penalty). Role: resource pressure. Proxies: queue depth, P95 latency, Triton

Read the complete text