Position: The Most Expensive Part of an LLM should be its Training Data

Nikhil Kandpal, Colin Raffel · 2025

Read Position: The Most Expensive Part of an LLM should be its Training Data by Nikhil Kandpal, Colin Raffel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12427.txt.

Description

Nikhil Kandpal and Colin Raffel argue that the most expensive part of a large language model should be the human writing it learned from. Training runs burn money on chips, power, and engineering, yet the trillions of words in the training data came from people's unpaid labor. The pair priced that labor for sixty-four models released between 2016 and 2024, asking what the datasets would cost to produce from scratch.

Even with cautious wage assumptions, the data dwarfed the training bill, by factors from ten to a thousand. That gap, they argue, is a real financial and moral liability for model providers. The paper closes by sketching research directions that could pay data creators fairly without breaking the economics of model building.

Catalog record

CREATORNikhil Kandpal, Colin Raffel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12427

Browse the full library · About the Tower of Babel