Deforking the World of Code: A Project-Provenance Map that Recovers Cross-Forge Fork Families that Platform Graphs Cannot See
Audris Mockus · 2026
Read Deforking the World of Code: A Project-Provenance Map that Recovers Cross-Forge Fork Families that Platform Graphs Cannot See by Audris Mockus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29550.txt.
Introduction
Mockus releases a deforking map covering the World of Code, a vast collection of open-source repositories. A fork copies a project's git history, so one commit surfaces in many repositories at once. Any popularity or spread measure counted over raw repositories is therefore inflated, sometimes by orders of magnitude. Platform fork graphs also miss families that span several hosting services.
His map collapses each raw repository into the deforked project it belongs to, using tens of millions of shared-commit groups. Simple merging over-connects unrelated software into giant clusters, so capped variants bound cluster size and remove boilerplate hub bridges. Validated against GitHub's declared fork graph rebuilt from archive events, the map agrees on nearly all shared edges. The disagreements expose fork families that GitHub's graph cannot represent, including families that span several hosts.
Catalog record
| CREATOR | Audris Mockus |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29550
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deforking the World of Code: A Project-Provenance Map that Recovers Cross-Forge Fork Families that Platform Graphs Cannot See Audris Mockus Abstract . Forks share git history, so a commit surfaces in many repositories and any spread- or popularity-based measure over raw repositories is inflated by orders of magnitude. We release a curated deforking map for the World of Code (WoC) version V2604 : p2PFull , which collapses every raw repository p p into the deforked project P P to which it belongs, built from the global shared-commit relation ( 51.79 51.79 M shared-commit groups) via a hub-node star encoding and parallel Louvain clustering, plus capped variants ( cap250 / cap500 ) that bound mega-cluster size. The naive shared-history union over-merges: the project graph welds unrelated software into giant clusters (largest uncapped cluster 861,948 861{,}948 repositories, bridged by shared-commit groups as large as 267,200 267{,}200 ), for the same structural reason author-identity graphs do. A cheap size cap removes the boilerplate-hub bridges; a structural-bridge diagnostic, the cut that dissolved the analogous author mega-cluster, run here but deliberately not applied , shows the post-cap residual is genuine vendored history , robust to the cut, so we leave it intact. We validate the map against GitHub’s declared fork graph reconstructed from GHArchive ForkEvent s, finding 99.01 % 99.01\% edge agreement conditional on both repositories being in WoC. Disagreements fall into two classes: a completeness byproduct (edges GitHub asserts but WoC has not ingested) and the central contribution, WoC-only fork families that GitHub’s platform graph cannot represent, including 5.41 % 5.41\% multi-forge families and 1.51 % 1.51\% whose fork root is not on GitHub. We additionally release a refreshed fork-exclusion list ( 134.1 134.1 M children, 3.4 × 3.4\times the GHTorrent-era 39.5 39.5 M) and a detached-fork inventory ( 455,550 455{,}550 hard-detached edges; 240,441 240{,}441 genuine independent origins). All artifacts are a self-contained, independently hosted replication package keyed to the WoC V2604 collection. 1. Introduction Mining software repositories at global scale must contend with forking: a single upstream history is copied into thousands of derived repositories, so a commit, a file, or an author appears to span far more “projects” than it truly does. Left uncorrected, every spread-, reach-, and popularity-based signal (author breadth, project size, dependency fan-out) is inflated by the fork multiplicity. Deforking , the recovery of the canonical project a repository belongs to, is therefore a prerequisite construction for any global-scale analysis, and it is exactly as error-prone as author-identity disambiguation: the obvious shared-history union over-merges catastrophically, and the fix requires the same precision discipline. This paper releases a deforking map for the World of Code (WoC) V2604 collection, extending the original shared-commit deforking approach ( 6 ) to the current collection, and documents both its construction and its external validation. Our contributions are the released artifacts and four findings about them, recorded as a living experiment log (Exps. D1–D5): • The map and its capped variants ( p2PFull , cap250 / cap500 ): a repository \rightarrow project provenance map over 51.79 M shared-commit groups, mega-cluster controlled (Exp. D1). • A structural diagnostic showing the post-cap residual is genuine shared (vendored) history rather than a clustering artifact, robust to the cut (Exp. D2), with its measured effect on the spread signal (Exp. D3). • External validation against GitHub’s declared fork graph from GHArchive ForkEvent s: 99.01\% edge agreement conditional on both endpoints in WoC, plus a completeness byproduct (Exp. D4). • Cross-forge and detached-fork analysis : the WoC-only fork families invisible to any single platform’s graph, a refreshed fork-exclusion list, and a detached-fork inventory (Exp. D5). The dataset is a drop-in join key for any WoC-scale study and, because it is built from observed shared commits rather than declared platform metadata, it captures fork relationships that span forges and survive history detachment, the cases platform fork graphs miss. 2. Related Work The closest prior work is the original shared-commit deforking map for the World of Code ( 6 ) . It established the idea this paper builds on: because git commits are Merkle-chained and practically impossible to reproduce independently, two repositories that share a commit are almost certainly the same project, so linking repositories by shared commits and extracting communities recovers fork families. Working over an earlier WoC version (about 1.9 B commits and 116 M repositories), it found that the naive shared-commit union produces a 13.9 M-repository mega-cluster, attacked it by manually removing backup and multi-project repositories and dropping commits that span more than a thousand projects, substituted GitHub’s ultimate parent for declared forks, and ran Louvain community detection in R/iGraph (in chunks, to skirt iGraph’s long-vector limit), reducing the largest community to about 355 K repositories. It validated against later-retrieved GitHub fork parents and against a GitHub-fork-based deduplication dataset ( 8 ) . We build directly on that approach and extend it in five ways, each documented as an experiment below. (1) Scale and method : we deform the current WoC V2604 collection ( 5.87 B commits, roughly three times the original), clustering a hub-node star encoding of the shared-commit relation with parallel Louvain rather than the original’s chunked R/iGraph run, which could not hold the graph in memory at once. (2) A deterministic size cap : dropping shared-commit groups larger than a cap C inside the graph builder removes the boilerplate hubs that bridge unrelated families, recall-safely and reproducibly. This generalizes the original’s manual bad-project list and its ad hoc removal of commits spanning more than a thousand repositories into a single tunable, in-builder parameter (Exp. D1). (3) A structural diagnosis of the residual: a sampled-betweenness analysis shows the post-cap residual is genuine vendored history (Chromium in Qt in Debian), not a clustering artifact, and so should be kept rather than cut (Exp. D2). (4) Cross-forge framing : because the map clusters by shared commits and not by any forge’s fork pointer, it recovers fork families that span forges and families rooted off GitHub entirely, which we quantify (Exp. D5). (5) Event-stream validation at full scale : rather than slowly polling the GitHub fork API, we reconstruct GitHub’s declared fork graph from the GHArchive ForkEvent stream and cross-check the entire map against it, turning the disagreement into a usable cross-forge and ingestion-completeness signal (Exp. D4). The declared-fork resources we validate against are platform-metadata graphs. GHTorrent mirrors GitHub’s event API and exposes parent–child fork edges and a fork-exclusion set ( 1 ; 2 ) ; GitHub’s own REST/GraphQL fork API offers the same relation live; and the deduplication dataset ( 8 ) packages GitHub’s ultimate-parent forks as a cleaning resource. All are metadata graphs: an edge exists only if a fork was created through a forge’s UI, on that forge, within the recorded window. Kalliamvakou et al. ( 3 ) document the resulting perils: many declared forks are empty, never-pushed, or otherwise inactive, and the recorded chain root need not be the true upstream, which our edge-level partition (Exp. D4) quantifies directly. Our map is complementary and commit-based: following World of Code ( 4 ; 5 ) , we recover the canonical project from observed shared commits , so a fork is detected whenever two repositories share git objects, regardless of which forge hosts them or whether the fork