Gradients: When Markets Meet Fine-tuning - A Distributed Approach to Model Optimisation
Christopher Subia-Waud · 2025
Read Gradients: When Markets Meet Fine-tuning - A Distributed Approach to Model Optimisation by Christopher Subia-Waud (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07940.txt.
Introduction
Christopher Subia-Waud, writing alone in 2025, describes Gradients, a market-style system for tuning AI models. Fine-tuning a model well takes the right settings, and big cloud platforms often pick poor ones. Gradients runs a contest instead: many independent workers race to find the best settings. Winners earn rewards tied to how well their models score.
The system runs on the Bittensor network, a shared platform for such contests. In one hundred eighty tuning tasks, Gradients beat three big platforms every time. It also beat a fourth platform in about five of six tasks. Average gains over the commercial tools reached about forty-two percent. The results suggest paid contests can find settings that single in-house methods miss.
Catalog record
| CREATOR | Christopher Subia-Waud |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-07940
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gradients: When Markets Meet Fine-tuning - A Distributed Approach to Model Optimisation Christopher Subia-Waud Abstract Foundation model fine-tuning faces a fundamental challenge: existing AutoML platforms rely on single optimisation strategies that explore only a fraction of viable hyperparameter configurations. In this white paper, We introduce Gradients, a decentralised AutoML platform that transforms hyperparameter optimisation into a competitive marketplace where independent miners compete to discover optimal configurations. Economic incentives align individual exploration with collective optimisation goals, driving systematic investigation of hyperparameter regions that centralised methods miss. We evaluate our approach across 180 controlled experiments spanning diverse model architectures (70M to 70B parameters) and task types. Gradients achieves an 82.8% win rate against HuggingFace AutoTrain and 100% against TogetherAI, Databricks, and Google Cloud, with mean improvements of 11.8% and 42.1% respectively. Complex reasoning and retrieval tasks show particularly strong gains of 30-40%, whilst diffusion models achieve 23.4% improvements for person-specific generation. These results demonstrate that competitive, economically-driven approaches can systematically discover superior configurations that centralised AutoML consistently miss. Figure 1 : Statistical analysis by task type. Four-panel analysis: box plots show improvement distributions, bar charts display win rates and mean improvements with standard errors, summary table provides exact values. 1 The open challenges in model fine-tuning. Foundation model fine-tuning involves optimisation across multiple interdependent dimensions: task type, model architecture, dataset size, and time constraints [ 41 ] . A 70M parameter model fine-tuned for mathematical reasoning with limited data requires different strategies than a 70B model trained for translation tasks with extensive datasets. Time constraints further complicate the picture—optimal configurations for 2-hour training windows differ from those for 8-hour allocations. Current AutoML platforms [ 37 , 13 ] produce results across these diverse scenarios, yet our empirical evaluation across 180 experiments reveals systematic performance gaps, suggesting that task-specific optimisation opportunities remain undiscovered. The challenge lies in creating systems that can adapt optimisation strategies to these multifaceted problem characteristics. Economic incentives offer a solution [ 11 , 8 ] : when optimisation success directly determines rewards, agents develop approaches tailored to the specific combination of task type, model scale, dataset properties, and resource constraints. Such incentive alignment naturally leads to decentralised approaches [ 24 , 17 ] . By distributing the exploration task across multiple independent agents, each with unique strategies and computational resources, decentralised systems can simultaneously investigate diverse approaches that would otherwise remain unexplored. This parallel search capability scales with the number of participants, creating an explorative capacity that exceeds what centralised systems can achieve with equivalent resources. However, mere distribution of computation is insufficient; the key innovation lies in establishing economic incentives that align individual agent behaviour with collective optimisation goals. When agent rewards are directly tied to discovering superior configurations, the system naturally evolves toward increasingly sophisticated optimisation strategies [ 20 ] . This evolutionary pressure creates a form of collective intelligence that continuously adapts to the specific characteristics of each fine-tuning task, rather than applying standardised approaches. The theoretical foundation of this approach combines distributed optimisation [ 23 ] , empirical risk minimisation, and tournament theory [ 26 ] , establishing a framework where properly structured competition can efficiently navigate complex optimisation landscapes that resist traditional methods. In this paper, we present Gradients, a decentralised AutoML platform built on the Bittensor network [ 32 ] that implements these theoretical principles to address the fine-tuning challenges of foundation models. We make several contributions: First, we formulate a mathematical scoring framework that effectively ranks model submissions based on their performance across both test and synthetic evaluation datasets, with appropriate mechanisms to prevent overfitting and ensure fair competition. Second, we develop an economic incentive structure that aligns individual miner rewards with overall optimisation quality, creating a self-reinforcing cycle of innovation in hyperparameter exploration. Third, we present empirical results from extensive comparative testing against leading centralised AutoML platforms, demonstrating consistent and significant performance improvements across diverse model types and application domains. Fourth, we analyse the dynamics of the system, showing how the competitive marketplace accelerates the adoption of novel optimisation techniques and enables more efficient allocation of computational resources compared to conventional approaches. Through these contributions, we demonstrate the practical effectiveness of decentralised, economically-driven approaches to hyperparameter optimisation and provide empirical evidence for their advantages, with particular application to the fine-tuning of Large Language Models and diffusion models. 2 Related Work Hyperparameter Optimisation and AutoML. The field of automated hyperparameter optimisation has progressed from simple grid and random search methods to sophisticated multi-fidelity approaches. Bayesian optimisation frameworks like Hyperband [ 23 ] and BOHB [ 12 ] combine principled uncertainty estimation with early stopping mechanisms to efficiently navigate large hyperparameter spaces. These methods achieve computational efficiency through bandit-based resource allocation, but remain fundamentally centralised in their coordination mechanisms. Recent AutoML platforms including AutoKeras [ 21 ] and Auto-Sklearn [ 13 ] have democratised access to these techniques, yet they still rely on single optimisation strategies that may miss task-specific optima. HuggingFace AutoTrain [ 37 ] represents the current state-of-the-art in no-code model fine-tuning, providing automated hyperparameter optimisation across multiple modalities but using centralised search strategies. Distributed and Competitive Optimisation. Population-based training (PBT) [ 20 ] introduced competitive dynamics to neural network optimisation, where populations of models compete and evolve hyperparameters during training. This approach demonstrates how competition discovers superior configurations through exploitation and exploration mechanisms reminiscent of evolutionary algorithms [ 10 ] . Parallel to this, federated learning [ 24 ] established principles for coordinating distributed model training without centralising data, achieving communication efficiency through local computation and periodic aggregation. However, these approaches lack economic incentive mechanisms to align individual participant behaviour with collective optimisation goals. Parameter-Efficient Fine-tuning. The emergence of foundation models has necessitated new approaches to efficient adaptation. LoRA [ 19 ] and its variants inject trainable low-rank matrices into pre-trained transformers, reducing trainable parameters by orders of magnitude whilst preserving performance. AdaLoRA [ 42 ] extends this approach with adaptive rank allocation, whilst QLoRA [ 7 ] combines quantisation with low-rank adaptation for memory efficiency. These parameter-efficient methods are widely used by AutoML LLM and diffusion training platforms [ 9 ] . Economic Mechanisms in Machine Learning. Recent work explores how economic incentives coordinate machine learning systems. Differentiable economics [ 11 ] employs neural networks to automatically design optimal auction mechanisms, demonstrating how learning systems embed economic principles. Blockchain-based coordination mechanisms [ 17 , 34 ] show promise for incentivising distributed computation, whilst tournament theory provides mathematical foundations for competition-based optimisation [ 26 ] . Incentive mechanism design for distributed coded machine learning [ 8 ] establishes optimal reward structures for heterogeneous computational contributors. However, researchers have largely left unexplored the application of these economic principles to automated hyperparameter optimisation. Decentralised Machine Learning Networks. Emerging platforms like Bittensor [ 32 ] demonstrate how cryptoeconomic incentives coordinate distributed machine learning tasks. These networks employ token-based rewards to align individual computational contributions with network-wide objectives, creating sustainable ecosystems for collaborative AI development. The Yuma Consensus mechanism [ 27 ] provides robust peer-ranking systems resistant to collusion, whilst maintaining decentralised coordination. Our work builds on these foundations by specifically applying economic coordination mechanisms to the hyperparameter optimisation problem, demonstrating how competitive markets discover superior model configurations. Foundation Model Fine-tuning and Evaluation. Recent