entralized Storage Designed to Serve

Guy Goren, Andrew Hariri, Timothy D. R. Hartley, Ravi Kappiyoor, Alexander Spiegelman, David Zmick · 2025

Read entralized Storage Designed to Serve by Guy Goren, Andrew Hariri, Timothy D. R. Hartley, Ravi Kappiyoor, Alexander Spiegelman, David Zmick (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19233.txt.

Introduction

Guy Goren and co-authors present Shelby, a decentralized storage protocol built for real workloads, in a paper from 2025. The head gives the full title, Shelby: Decentralized Storage Designed to Serve. Older decentralized systems are too slow and costly for video, analytics, or AI training. Web3 apps thus lean on centralized clouds anyway.

Shelby borrows Web2 lessons instead. Control and data planes split apart. Erasure coding keeps copies cheap with little repair traffic. A dedicated backbone links the serving and storage nodes. Reads are paid, which rewards fast, honest nodes. A new auditing scheme checks that data stays put. The pitch is decentralization that finally performs like the services it hopes to replace.

Catalog record

CREATORGuy Goren, Andrew Hariri, Timothy D. R. Hartley, Ravi Kappiyoor, Alexander Spiegelman, David Zmick
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-19233

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

entralized Storage Designed to Serve Guy Goren, Andrew Hariri, Timothy D. R. Hartley, Ravi Kappiyoor, Alexander Spiegelman, David Zmick Abstract Existing decentralized storage protocols fall short of the service required by real-world applications. Their throughput, latency, cost-effectiveness, and availability are insufficient for demanding workloads such as video streaming, large-scale data analytics, or AI training. As a result, Web3 data-intensive applications are predominantly dependent on centralized infrastructure. high-performance decentralized storage protocol designed to meet demanding needs. It achieves fast, reliable access to large volumes of data while preserving decentralization guarantees. The architecture reflects lessons from Web2 systems: it separates control and data planes, uses erasure coding with low replication overhead and minimal repair bandwidth, and operates over a dedicated backbone connecting RPC and storage nodes. Reads are paid, which incentivizes good performance. o introduces a novel auditing protocol that provides strong cryptoeconomic guarantees without compromising performance, a common limitation of other decentralized solutions. The result is a decentralized system that brings Web2-grade performance to production-scale, read-intensive Web3 applications. 1 1 1 Disclaimer: For convenience and consistency, this paper describes Shelby in the present tense. As a result, this paper includes certain forward-looking statements based on the Shelby design and expectations of the authors as of the date of this paper, but please note that future facts or operation of the Shelby protocol may differ from the content expressed herein. 1 Introduction Web3 has made major strides in decentralized payments and trading use cases. Despite this progress, the ecosystem still lacks the core infrastructure needed to scale: decentralized cloud services. Without high-performance alternatives to cloud storage, networking, and compute, Web3 applications hit hard scalability limits, constraining both what they can do and who they can serve. A critical missing piece is hot storage : infrastructure that can support low-latency, high-throughput reads. Use cases like video streaming, real-time collaboration, and AI inference all rely on fast, reliable access to large datasets. While protocols like Filecoin [ 5 ] , Arweave [ 1 ] , Walrus [ 13 ] and Celestia [ 7 ] support archival or cold storage, none can meet the performance requirements of dynamic, read-heavy applications. The barrier is not just technical - it is economic. Reads are bandwidth-intensive, latency-sensitive, and difficult to meter in a decentralized system. Without a viable economic model for rewarding high-performance reads, most systems avoid the problem entirely, focusing instead on infrequent writes or cold data storage. As a result, truly decentralized applications that depend on fast reads continue to rely on centralized infrastructure. To ground this challenge in a real-world context, consider video streaming - a canonical example of a read-heavy workload where latency and throughput are paramount. Supporting high-quality 4K video playback requires sustained throughput of at least 40 Mbps, typically delivered in 10 MiB chunks with minimal startup delay. No existing decentralized storage network satisfies this read-throughput requirement (see section 1 for a partial list). Beyond media, modern AI workflows such as retrieval-augmented generation (RAG) systems depend on low-latency access to diverse, high-quality datasets. These models are rapidly moving from research to real-world use: clinicians training diagnostic tools on medical images, governments building domain-specific assistants, and robotics teams teaching home agents new tasks. The expected utility of these systems is large, but realizing it requires two conditions: data must be easy to monetize with fair compensation, and access to that data must be fast and efficient. A data explosion is underway, and while Web3 is uniquely positioned to preserve user control, current infrastructure cannot meet the performance and economic demands of this shift. decentralized storage network purpose-built for high-performance reads. It combines cryptoeconomic incentives with engineering principles drawn from high-performance Web2 systems to deliver fast, reliable access to data without sacrificing decentralization. bines a novel auditing protocol, efficient erasure coding, and a dedicated fiber network to support Web2-grade performance at Web3 trust levels. Its architecture establishes a strong game-theoretic equilibrium in which all participants - storage providers, RPC nodes, and clients - are incentivized to behave honestly and serve data quickly. At its core, the system is built on six tightly integrated pillars, each contributing to its decentralized performance and economic sustainability. Paid reads - aligned performance incentives. When users pay for a service, they expect it to be high quality. By requiring payment for reads, gns the incentives for users and service providers. Everyone in the ecosystem wants more data to flow: users want a great experience streaming high-quality video or accessing large files and service providers want to serve as much data as they can. Dedicated fiber network - high-performance network infrastructure. Unlike the public internet - where congestion, variable latency, and unpredictable routing can degrade performance - a dedicated backbone provides h consistent bandwidth and low-latency guarantees critical for real-time data access. By deploying storage and RPC nodes directly onto this network, ers users sub-second access latency and reduces bandwidth costs dramatically - two requirements essential for hot storage and applications like streaming and real-time collaboration. Such a network is critical to reach performance comparable to Web2 cloud providers. Battle-tested software stack - proven infrastructure for low-latency storage systems. oftware stack is built on engineering principles honed through years of experience developing storage and compute systems for Jump Trading Group’s high-performance quantitative research infrastructure and Meta’s global-scale platforms. This includes high-performance I/O pipelines, efficient concurrency primitives, and low level code optimizations. The result is a system architecture that can handle bursty workloads, maintain sub-second level responsiveness, and scale horizontally with minimal overhead. Efficient Coding Scheme - high durability with low storage overhead. A core challenge in decentralized storage is achieving cost efficiency without sacrificing durability. That is, reducing replication overhead while being able to tolerate and efficiently repair significant amounts of data corruption. To make this viable, s Clay codes [ 20 ] , an advanced erasure coding scheme that achieves theoretically optimal repair bandwidth (i.e., Minimum Storage Regenerating) without compromising reconstruction capability (i.e., Maximum Distance Separable). This balance is what enables perate with a much lower replication factor than typical Web3 storage networks. While many decentralized systems rely on full replication or less efficient erasure codes - often resulting in 5x to 8x overhead - ieves durability with a replication factor under 2x. This represents a major cost advantage and brings se to the 1.2x-1.4x replication range common in high-performance Web2 storage systems. Incentive-Compatible auditing - economic security without performance trade-offs uditing protocol is a hybrid design that combines high-frequency internal audits with low-frequency, cryptographically enforced on-chain verification. Storage providers are continuously challenged to prove possession of random data samples, while their audit behaviors are themselves audited on-chain. This “audit-the-auditor” mechanism allows ignificantly reduce the auditing cost by performing most audits internally, while ensuring that no participant can game the system through collusion or apathy. Importantly, this structure is fully incentive-compatible: honest participation maximizes rewards, while misbehavior leads to slashing and revenue loss. This ensures a strong equilibrium where nodes store data reliably, respond to reads quickly, and verify one another in a decentralized yet economically rational way. Aptos Blockchain - fast and reliable coordination layer. s the Aptos blockchain as its coordination and settlement layer. Aptos offers high transaction throughput, low finality times, and a resource-efficient execution model,

Read the complete text