Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components
Ling Zhan, Xiaoyao Yu, Tao Jia · 2026
Read Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components by Ling Zhan, Xiaoyao Yu, Tao Jia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24558.txt.
Introduction
Ling Zhan, Xiaoyao Yu, and Tao Jia argue that measurement-to-dataset pipelines in AI for science do inference work, not fixed data. Treating shared data as given freezes an observation model and hides uncertainty over pipeline choices. They name three failure modes: hidden hypothesis spaces, uncertified transportability, and ungoverned multiplicity. A brain-science audit found a survival rate of roughly 0.0004 percent under a cross-dataset stability check, and the authors call for pipelines to become computable, auditable objects.
Catalog record
| CREATOR | Ling Zhan, Xiaoyao Yu, Tao Jia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78909 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24558
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position: AI for Science Should Treat Measurement-to-Dataset Pipelines as Inference Components Ling Zhan, Xiaoyao Yu, Tao Jia Abstract AI for Science (AI4Science) workflows often treat the released dataset as a fixed interface to the underlying system. However, in domains relying on indirect observation , the learner observes a derivative representation produced by multi-stage measurement, reconstruction, and preprocessing pipelines. We argue that these measurement-to-dataset pipelines are inference components: treating their outputs as “given data” freezes an observation model and obscures uncertainty over feasible pipeline choices. We identify three failure modes arising from this “frozen lens”: (C1) hidden hypothesis space , where the released dataset does not specify the pipeline configuration or its validity conditions; (C2) uncertified transportability , where a pipeline may be documented but its regime of validity is untested, so failures under distribution shift cannot be adjudicated; (C3) ungoverned multiplicity , where many defensible pipelines exist and dispersion is real but not propagated into uncertainty-aware evidence. We stress-test these claims with a large-scale neuroscience empirical audit, finding a survival rate of ≈ 0.0004 % \approx 0.0004\% under a cross-dataset stability criterion. We call on the AI4Science community to make pipelines computable inference objects via domain-specific Computable Observation Frameworks. This shift enables quantifying pipeline adequacy and stability, converting implicit implementation choices into auditable, reproducible, and cumulative scientific evidence. † † affiliationnotice: Equal contribution 1 Introduction AI for Science (AI4Science) is often framed around benchmark-driven predictive objectives, yet the epistemic aims of science are broader: hypothesis generation, theory building, and mechanistic explanation tied to the underlying system ( 123 ; 78 ) . A growing literature integrates modern machine learning (ML) into discovery workflows across disciplines, from multi-messenger astrophysics to computational psychology ( 126 ; 121 ; 30 ; 60 ) . However, in many fields, research findings are mined from indirect observation : the inference of a scientific target from proxy measurements via a nontrivial observation model, effectively treating data acquisition as an inverse problem ( 109 ; 85 ) . For example, in radio interferometry, images are reconstructed from visibilities via calibration and deconvolution choices ( 115 ) . The same issue also appears in structural biology and drug discovery, where downstream models often consume curated structural or bioactivity records whose apparent “data” status depends on deposition, assay, normalization, and filtering conventions ( 44 ) . Appendix A gives representative pipelines spanning five indirect-observation domains. Consequently, AI4Science under indirect observation must consider not only what to learn, but also which pipeline configuration defines the representation to be learned from. This is a difference of degree rather than kind. Standard ML also depends on preprocessing, but its inferential object usually centers on the mapping from representation to prediction on an agreed task interface. Preprocessing typically serves this objective as performance-oriented engineering ( 129 ; 124 ) . In indirect-observation AI4Science, configuration \phi specifies a multi-stage mapping from sensor measurements to a released representation, with consequential degrees of freedom that are rarely specified, constrained, or audited ( 49 ; 2 ) . Because AI4Science often aims to support mechanistic claims about an underlying system that is not directly observable, the configured pipeline functions as an observation model linking measurements to hypothesized latent structure. Downstream learning can optimize performance on the released interface, but it yields evidence for the claimed mechanism only conditional on \phi ( 5 ; 4 ) . Figure 1 summarizes this shift from predictive inference to scientific inference. Both settings may instantiate the same computational template, but when the target moves from prediction to mechanism discovery, P_{\phi} directly shapes how mechanisms are inferred and interpreted. In indirect-observation AI4Science, released representations depend on weakly identified defaults, inherited recipes, and tuning decisions that encode assumptions about signal and artifact ( 16 ; 17 ; 62 ) . Freezing these choices yields a frozen lens : an implicit inference component whose output is treated as an objective data fact ( 90 ; 66 ) . This can reward predictive proxies while hiding pipeline uncertainty from the learner, weakening mechanistic interpretation ( 73 ) . Figure 1 : Predictive inference versus scientific inference. The contrast is a difference of degree rather than kind: standard ML also depends on preprocessing. In indirect-observation AI4Science, however, a pipeline P_{\phi} transforms raw measurements x into a released representation z=P_{\phi}(x) , from which a downstream model produces \hat{y}=f_{\theta}(z) . Mechanistic claims about the latent target \psi are therefore conditional on \phi . The inset gives the EEG audit instantiation. Our position: Pipelines are components of inference in AI4Science. They should be audited and evaluated under scientific validity criteria, not frozen as dataset definitions. Uncertainty over admissible observation models should be quantified and carried forward as cumulative evidence, rather than collapsed to a point choice. We articulate this position through three failure modes. (C1) Hidden hypothesis space: the released representation hides the pipeline-indexed space of mechanisms that could have been tested. (C2) Uncertified transportability: a documented pipeline rarely comes with a tested regime of validity across measurement settings. (C3) Ungoverned multiplicity: multiple defensible pipelines remain, but their dispersion is rarely propagated into uncertainty-qualified evidence. The paper proceeds by characterizing how pipelines become dataset interfaces (Section 2 ), deriving three failure modes (Section 3 ; Appendix B ), auditing EEG connectomics under admissible pipeline variation (Section 4 ; Appendix C ), addressing common objections (Section 5 ), and proposing computable observation frameworks (Section 6 ). 2 The Frozen Lens in AI4Science This section characterizes how current AI4Science workflows turn observation pipelines into dataset interfaces. Across pipeline-first, differentiable, and foundation-model paradigms, what travels across labs is often a derivative representation rather than raw measurement. Table 1 summarizes how each paradigm fixes, learns, or inherits a pipeline configuration \phi before downstream learning begins. Table 1: Bundling pipeline into “data” in current AI4Science workflows. Community-facing interfaces typically fix an observation model via a pipeline P_{\phi} (explicit, inherited, or learned-then-frozen). \phi indexes pipeline choices. Paradigm Typical Instantiations Practical Interface Role / Status of \phi Pipeline-first (classical) Processed-derivative benchmarks ( 46 ; 7 ) . Standardized toolchains ( 65 ; 94 ; 48 ) . A released representation (often with a reference pipeline/QC recipe) becomes the object of reuse and benchmarking. Explicit and fixed: \phi is selected upstream and executed once to produce the representation. Hybrid / Differentiable (end-to-end) Learnable front-ends ( 108 ) ; Differentiable operators ( 39 ) ; Unrolled physics-based modules ( 1 ) . A deployed end-to-end system (front-end + learner/reconstructor) with a standard inference API. Learned then fixed: \phi is optimized during training, then deployed as a fixed front-end. Model-first (foundation models) Curated pretraining data with fixed reconstruction/harmonization conventions ( 19 ; 111 ; 105 ) . A pretrained model and its representation space (embeddings, generative priors, or task adapters). Implicit and inherited: \phi is encoded in pretraining curation and reconstruction conventions. 2.1 The Evolution of the Frozen Lens In the common AI4Science regime, the data–model interface is shaped by modularity. Domain practitioners in computational chemistry, astrophysics, and neuroimaging apply established toolchains (e.g., RDKit , Astropy , MNE-Python ) to raw measurements or primary observational products to produce model-ready representations ( 53 ; 63 ; 22 ) . This separation is practical and reinforced by processed-derivative benchmarks, where the