Estimating Absolute Web Crawl Coverage from Longitudinal Set Intersections

Michael Paris, Grigori Paris, Fabian Baumann · 2026

Read Estimating Absolute Web Crawl Coverage from Longitudinal Set Intersections by Michael Paris, Grigori Paris, Fabian Baumann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15416.txt.

Description

Paris and co-authors estimate how much of its target space a web crawl actually covers, using only the archive's own history. Their trick reads the overlap of URL sets between one crawl and the next. A simple urn process fits those overlaps well. Linear regression then recovers the urn parameters and the absolute coverage. On fifteen semi-annual crawls of the German Academic Web, coverage came out near 46 percent. The method needs no external ground truth and fits any repeated focused crawl.

Catalog record

CREATORMichael Paris, Grigori Paris, Fabian Baumann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23187
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15416

Browse the full library · About the Tower of Babel