A Scalable Crawling Algorithm Utilizing Noisy Change-Indicating Signals DOI: XXXXXXX.XXXXXXX Conference: Proceedings of the ACM Web Conference 2025; April 28–May 2, 2025; Sydney, NSW, Australia. ISBN: 978-1-4503-XXXX-X/18/06 CCS: World Wide Web CCS: Web searching and information discovery CCS: Web s
Róbert Busa-Fekete, Julian Zimmert, András György, Linhai Qiu, Tzu-Wei Sung, Hao Shen, Hyomin Choi, Sharmila Subramaniam, Li Xiao · 2025
Read A Scalable Crawling Algorithm Utilizing Noisy Change-Indicating Signals DOI: XXXXXXX.XXXXXXX Conference: Proceedings of the ACM Web Conference 2025; April 28–May 2, 2025; Sydney, NSW, Australia. ISBN: 978-1-4503-XXXX-X/18/06 CCS: World Wide Web CCS: Web searching and information discovery CCS: Web s by Róbert Busa-Fekete, Julian Zimmert, András György, Linhai Qiu, Tzu-Wei Sung, Hao Shen, Hyomin Choi, Sharmila Subramaniam, Li Xiao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02430.txt.
Description
Robert Busa-Fekete and co-authors studied web crawling in two thousand twenty-five. Crawlers copy web pages so search tools can show fresh results. Bandwidth is limited, so a crawler must choose which pages to revisit. Pages send hints when their content changes, but those hints can be wrong. Some hints fire when nothing changed, and some changes send no hint.
The authors built a large-scale method that uses these noisy hints wisely. It rests on models of how often pages change and get requests. The method aims to keep copies fresh for all pages, not just busy ones. Their tests show hint-aware crawling beats schedules that ignore hints. The work matters for search engines that must track a fast-changing web.
Catalog record
| CREATOR | Róbert Busa-Fekete, Julian Zimmert, András György, Linhai Qiu, Tzu-Wei Sung, Hao Shen, Hyomin Choi, Sharmila Subramaniam, Li Xiao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-02430