Surrogate Representation Inference for Noisy Text and Image Annotations
Kentaro Nakamura · 2025
Read Surrogate Representation Inference for Noisy Text and Image Annotations by Kentaro Nakamura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12416.txt.
Introduction
Machine labels for photos and essays are cheap, but statistics built on them are biased, and fixing the bias usually demands some error-free human labels. In a solo 2025 paper, Kentaro Nakamura proposes Surrogate Representation Inference, which needs none, as long as human coders judge only the raw material itself.
A neural network learns a compact representation of the text or image that keeps the needed surrogacy property intact. With several human annotations available, the method can even correct steady human measurement error. In simulations and a real application, standard errors fall by more than half at moderate classifier accuracy.
Catalog record
| CREATOR | Kentaro Nakamura |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 171 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-12416
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Surrogate Representation Inference for Noisy Text and Image Annotations Thanks: I thank Naijia Liu, Soichiro Yamauchi, and the members of Imai research group for helpful comments. I especially thank Kosuke Imai for his continuous encouragement and detailed comments. Kentaro Nakamura Abstract As researchers increasingly rely on machine learning models and LLMs to annotate unstructured data, such as texts or images, various approaches have been proposed to correct bias in downstream statistical analysis. However, existing methods tend to yield large standard errors and require some error-free human annotation. In this paper, I introduce Surrogate Representation Inference (SRI), which assumes that unstructured data fully mediate the relationship between human annotations and structured variables. The assumption is guaranteed by design provided that human coders rely only on unstructured data for annotation. Under this setting, I propose a neural network architecture that learns a low-dimensional representation of unstructured data such that the surrogate assumption remains to be satisfied. When multiple human annotations are available, SRI can further correct non-differential measurement errors that may exist in human annotations. Focusing on text-as-outcome settings, I formally establish the identification conditions and semiparametric efficient estimation strategies that enable learning and leveraging such a low-dimensional representation. Simulation studies and a real-world application demonstrate that SRI reduces standard errors by over 50% when machine learning prediction accuracy is moderate and provides valid inference even when human annotations contain non-differential measurement errors. Keywords: Text-as-Data, Representation Learning, Semiparametric Statistics, Deep Learning, Double Negative Control 1 Introduction With a wide availability of texts and images, the use of these unstructured data for statistical inference has become increasingly common in many fields, including political science (e.g. Grimmer et al., 2022 ; Stukal et al., 2022 ; Wratil et al., 2023 ), economics (e.g. Gentzkow et al., 2019 ; Hassan et al., 2019 ; Ash and Hansen, 2023 ), and sociology (e.g. Kozlowski et al., 2019 ; Macanovic, 2022 ). While researchers use machine learning and large language models (LLMs) to annotate such data, their predictions contain errors that invalidate the downstream statistical inference. As a result, various approaches have been proposed to correct the bias in the downstream analysis by combining the human annotations and machine learning or LLM predictions (e.g. Fong and Tyler, 2021 ; Angelopoulos et al., 2023 ; Egami et al., 2023 ; Egami et al., 2024 ; Zrnic and Candès, 2024 ; Carlson and Dell, 2025 ). However, the existing methodology tends to yield large standard errors, especially when the quality of machine learning predictions is moderate, and assumes human annotations contain no error. In this paper, I introduce a new methodological framework, Surrogate Representation Inference (SRI), which improves efficiency and yields statistically valid estimates even when human annotations contain non-differential measurement errors. SRI improves efficiency based on the assumption that unstructured data work as a surrogate variable, fully mediating the relations between human annotations and structured variables of interest, such as predictors. This assumption is guaranteed by design if human coders rely only on texts and images for the annotation task and do not directly look at other relevant structured variables. Since unstructured data are high-dimensional, I propose using low-dimensional representations that still satisfy the surrogacy assumption. I refer to these as surrogate representations . Because surrogate representation can leverage surrogate structure, the proposed SRI methodology can improve efficiency. To estimate surrogate representation and perform subsequent statistical inference, I introduce a neural network architecture that jointly predicts human annotations and structured variables. As a concrete example, I demonstrate the proposed SRI methodology in the setting where the text-based variable, such as sentiments or topics, serves as an outcome. I show that the covariate-adjusted outcome mean (i.e., the average value of the text-based variable for each level of the predictors after adjusting for the set of control variables) is nonparametrically identified using surrogate representation. This surrogate representation can be estimated with the proposed neural network architecture. Drawing on the literature of semiparametric statistics (e.g. Tsiatis, 2006 ; Chernozhukov et al., 2018 ; Kennedy, 2023 ), I derive the semiparametric efficient estimation strategies tailored to the proposed neural network architectures and formalize the conditions under which the SRI estimator is valid and efficient. The SRI framework can further adjust measurement errors in human annotations when multiple human annotations are available. Specifically, I show that when the measurement error in human annotations is non-differential (i.e., the measurement error does not depend on the true annotation or the contents of texts), the quantity of interest can be nonparametrically identified and estimated from multiple annotations using the literature of double negative control (e.g., Kuroki and Pearl, 2014 ; Tchetgen et al., 2020 ; Zhou and Tchetgen, 2024 ). Since perfect human annotations are not required, once researchers can create a solid coding rule, they can then leverage existing crowd-sourcing platforms, such as Amazon Mechanical Turk and CrowdFlower, to generate human-annotated labels with relative ease. I finally apply the proposed SRI methodology to the existing research on the framing of immigrants in the U.S. Congressional speeches ( Card et al., 2022 ) . I compare the proposed SRI estimator with the existing bias correction algorithm ( Angelopoulos et al., 2023 ; Egami et al., 2023 ; Egami et al., 2024 ) that leverages both machine learning predictions and human annotations. I find that the direct use of machine learning predictions inflates the partisan difference in the framing of immigrants. Both SRI and the existing bias-correction approach yield consistent point estimates, but because tone classification is challenging, SRI reduces the standard errors by roughly one-third. In Appendix S3 , I also validate the performance of the proposed estimator through simulation studies, in which I systematically vary the quality of the machine learning predictions. The findings reveal that SRI outperforms the existing approach across all settings except the case that the machine learning predictions are already almost perfect (above 90% accuracy), with the efficiency gains becoming particularly pronounced when proxy accuracy is moderate (e.g., below 85%). I further find that when human annotations contain classification errors, the existing bias-correction approach, which relies on the assumption of perfect human annotations, suffers from substantial bias and improper confidence interval coverage, but the proposed SRI estimator with multiple human labels gives valid estimates with the smaller bias and root mean square error (RMSE) compared to the existing bias-correction methodology. The remainder of this paper is organized as follows. Section 2 presents a motivating example drawn from the actual published research. Section 3 formalizes the proposed methodology with error-free human annotations. Section 4 then relaxes this assumption and deals with more realistic setting in which humans annotations contain non-differential measurement errors. Section 5 then applies the proposed method to the existing study introduced in the motivating example. Finally, Section 6 concludes by summarizing the advantages and limitations of SRI and outlining directions for future research. Related Literature. There is a growing body of literature that aims to conduct statistical inference with a limited amount of ground-truth labels by leveraging both labeled data and machine learning predictions (e.g., Fong and Tyler, 2021 ; Angelopoulos et al., 2023 ; Egami et al., 2023 ; Egami et al., 2024 ; Zrnic and Candès, 2024 ; Carlson and Dell, 2025 ). Unlike the existing approaches,