Investigating the Association Between Text-Based Indications of Foodborne Illness from Yelp Reviews and New York City Health Inspection Outcomes (2023)
Eden Shaveet, Crystal Su, Daniel Hsu, Luis Gravano, [4pt] Columbia University, Department of Computer Science, New York, NY, USA {ems2349, ys3791}@columbia.edu {djhsu, gravano}@cs.columbia.edu · 2025
Read Investigating the Association Between Text-Based Indications of Foodborne Illness from Yelp Reviews and New York City Health Inspection Outcomes (2023) by Eden Shaveet, Crystal Su, Daniel Hsu, Luis Gravano, [4pt] Columbia University, Department of Computer Science, New York, NY, USA {ems2349, ys3791}@columbia.edu {djhsu, gravano}@cs.columbia.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16334.txt.
Introduction
Eden Shaveet and three co-authors test whether Yelp complaints can flag dirty restaurants before inspectors do. Foodborne illness is badly underreported through official channels, while diners narrate symptoms in reviews, so a classifier's illness signals across reviews might track real hygiene failures.
The team feeds reviews through a hierarchical attention classifier and compares its signals with New York City health inspection outcomes from 2023, aggregating at the census tract level and mapping spatial patterns. The honest answer is mostly negative: correlations with inspection scores are minimal, and tracts heavy in C-graded restaurants show no significant signal difference. The authors discuss why tract-level aggregation may dilute the effect and outline finer address-level analyses as the next step.
Catalog record
| CREATOR | Eden Shaveet, Crystal Su, Daniel Hsu, Luis Gravano, [4pt] Columbia University, Department of Computer Science, New York, NY, USA {ems2349, ys3791}@columbia.edu {djhsu, gravano}@cs.columbia.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16334
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating the Association Between Text-Based Indications of Foodborne Illness from Yelp Reviews and New York City Health Inspection Outcomes (2023) Eden Shaveet, Crystal Su, Daniel Hsu, Luis Gravano, [4pt] Columbia University, Department of Computer Science, New York, NY, USA {ems2349, ys3791}@columbia.edu {djhsu, gravano}@cs.columbia.edu Abstract Foodborne illnesses are gastrointestinal conditions caused by consuming contaminated food. Restaurants are critical venues to investigate outbreaks because they share sourcing, preparation, and distribution of foods. Public reporting of illness via formal channels is limited, whereas social media platforms host abundant user-generated content that can provide timely public health signals. This paper analyzes signals from Yelp reviews produced by a Hierarchical Sigmoid Attention Network (HSAN) classifier and compares them with official restaurant inspection outcomes issued by the New York City Department of Health and Mental Hygiene (NYC DOHMH) in 2023. We evaluate correlations at the Census tract level, compare distributions of HSAN scores by prevalence of C-graded restaurants, and map spatial patterns across NYC. We find minimal correlation between HSAN signals and inspection scores at the tract level and no significant differences by number of C-graded restaurants. We discuss implications and outline next steps toward address-level analyses. 1 Introduction Foodborne illness imposes substantial morbidity and economic burden globally, accounting for millions of cases of gastrointestinal disease each year and resulting in significant public health costs. Traditional surveillance systems primarily rely on clinical reports, laboratory confirmations, and regulatory inspections. While these systems are essential for official monitoring and response, they often suffer from delays and underreporting because affected individuals do not always seek medical attention or submit formal complaints. Consequently, many potential outbreaks go undetected until they become widespread. To address this limitation, social media data offer an alternative and complementary channel for detecting early signals of foodborne illness. User-generated platforms such as Yelp provide spontaneous, real-time accounts of dining experiences, some of which include descriptions of suspected illness after restaurant visits. The Adaptive Information Extraction from Social Media for Actionable Inferences in Public Health project leverages these data to augment existing surveillance mechanisms. In this collaboration between Columbia University and the New York City Department of Health and Mental Hygiene (NYC DOHMH), Yelp reviews are retrieved daily and analyzed using a Hierarchical Sigmoid Attention Network (HSAN) model [ 4 ] . The HSAN classifier assigns each review a probability score indicating its likelihood of describing a foodborne illness event. Reviews exceeding a predefined threshold are then transmitted to NYC DOHMH for further evaluation alongside traditional inspection workflows. This pipeline enables the near real-time monitoring of potential public health threats and helps prioritize inspections based on emerging online patterns. The present study investigates whether HSAN-derived social media signals align with official NYC DOHMH restaurant inspection outcomes at the Census tract level for 2023. By quantifying these relationships spatially and statistically, we aim to assess whether user-reported illness mentions can serve as a meaningful proxy for actual food safety conditions, potentially strengthening hybrid surveillance models that integrate machine learning with regulatory oversight. 2 Data and Methods 2.1 Data Sources This study integrated three primary datasets. Yelp restaurant reviews from establishments across the five boroughs of New York City were obtained through a private API partnership for the 2023 calendar year. Each record contained textual content, review date, user rating, restaurant metadata, and geographic coordinates. Official restaurant inspection results were sourced from the New York City Department of Health and Mental Hygiene (NYC DOHMH) Open Data portal, which provides both numeric inspection scores—where higher scores indicate poorer sanitary outcomes—and corresponding letter grades (A, B, or C). Census tract boundary shapefiles were downloaded from the U.S. Census Bureau’s TIGER/Line dataset to ensure a consistent geographic framework for aggregation. Restaurant addresses were geocoded using the OpenStreetMap (OSM) Nominatim service, and duplicate or incomplete records were removed before spatial joining to maintain data accuracy. 2.2 HSAN Scoring Each Yelp review was processed using a pre-trained Hierarchical Sigmoid Attention Network (HSAN) classifier [ 4 ] , a weakly supervised neural architecture designed to detect potential foodborne illness mentions in user-generated text. The HSAN model captures contextual information through a hierarchical attention mechanism, identifying relevant tokens and sentences while outputting a continuous probability score between 0 and 1 via a sigmoid activation layer. For each review r_{i} , the model produced a likelihood value p_{i} reflecting the probability that the review described a foodborne illness event. Reviews with p_{i}\geq 0.05 were retained for analysis, a threshold chosen to balance false positives and recall based on prior validation studies. Model inference was performed using Python 3.10 and the PyTorch framework on Google Colab. 2.3 Aggregation To enable spatial comparison between text-based illness signals and official inspection results, both datasets were aggregated at the Census tract level. Each geocoded restaurant was assigned to its corresponding tract polygon using a spatial join in GeoPandas. For every tract t_{j} , we computed the mean HSAN probability \overline{p_{j}} across all qualifying reviews and the mean inspection score \overline{s_{j}} across all inspected restaurants within the same tract. Tracts with fewer than three valid reviews or two inspections were excluded to reduce statistical instability arising from small sample sizes. All spatial processing, including coordinate transformations and boundary clipping, was performed using QGIS 3.34 and GeoPandas 0.14. 2.4 Analyses Spatial visualization and statistical testing were carried out to explore the relationship between social media–derived indicators and inspection outcomes. Choropleth maps of tract-level mean HSAN and inspection scores were created, along with a kernel-density heatmap weighted by HSAN probabilities to visualize concentrations of illness-related reviews. A bivariate choropleth map was also generated to highlight tracts where both indicators were simultaneously elevated. Pearson ( r ) and Spearman ( \rho ) correlation coefficients were calculated between \overline{p_{j}} and \overline{s_{j}} to assess both linear and rank-based associations, with separate analyses performed for each borough to evaluate spatial heterogeneity. Additionally, the distribution of tract mean HSAN scores was compared across three categories of restaurant environments—tracts containing 1–2, 3–4, or five or more C-graded restaurants—using the Kruskal–Wallis nonparametric test to determine whether statistically significant differences existed among groups. All analyses were conducted in Python 3.10 using pandas , scipy , and numpy for computation and visualization, with cartographic design completed in QGIS. This integrated workflow links machine learning–based social media signal extraction with regulatory food safety data, facilitating both quantitative assessment and visual exploration of potential associations between public online discourse and official inspection outcomes across New York City. 3 Results 3.1 Spatial Patterns Figure 2 illustrates the spatial distribution of foodborne-illness–related Yelp reviews and official inspection outcomes across New York City. The bivariate map combines both indicators, while the univariate panels display each metric individually along with the HSAN-weighted point density of reviews. Brooklyn shows numerous tracts with relatively high HSAN and inspection scores, indicating areas with frequent illness mentions as well as less favorable inspection results. In contrast, Manhattan and Queens display greater heterogeneity, with clusters of high HSAN activity that do not consistently correspond to poor inspection outcomes. Staten Island and the Bronx exhibit fewer Yelp reviews overall, which likely reflects differences in population density and restaurant volume rather than genuine absence of risk. These spatial patterns suggest that social media signals and inspection metrics vary considerably across boroughs, shaped by both demographic and