Uncertainty-Guided Expert-AI Collaboration for Efficient Soil Horizon Annotation DOI: XXXXXXX.XXXXXXX Conference: European Conference on Artificial Intelligence, CLEAR-AI Workshop: Collaborative Methods and Tools for Engineering and Evaluating Transparency in AI; October 25–30, 2025; Bologna, Italy

Teodor Chiaburu, Vipin Singh, Frank Haußer, Felix Bießmann · 2025

Read Uncertainty-Guided Expert-AI Collaboration for Efficient Soil Horizon Annotation DOI: XXXXXXX.XXXXXXX Conference: European Conference on Artificial Intelligence, CLEAR-AI Workshop: Collaborative Methods and Tools for Engineering and Evaluating Transparency in AI; October 25–30, 2025; Bologna, Italy by Teodor Chiaburu, Vipin Singh, Frank Haußer, Felix Bießmann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24873.txt.

Introduction

Teodor Chiaburu, Vipin Singh, Frank Hausser, and Felix Biessmann presented a workshop paper in twenty twenty-five on labeling soil layers with help from artificial intelligence. Experts read soil profiles and mark where one horizon ends and the next begins, but expert time is scarce. The authors use a model that describes soil profiles and add a method that states how unsure each prediction is. In a simulated setup, experts are called in only where the model is most unsure, within a fixed budget.

With the uncertainty step added, labeling for measured values became more efficient under the same budget. Results for category labels were about the same as without the step. The work matters for teams that must spend limited expert time where it helps most, rather than checking every model output.

Catalog record

CREATORTeodor Chiaburu, Vipin Singh, Frank Haußer, Felix Bießmann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24873

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uncertainty-Guided Expert-AI Collaboration for Efficient Soil Horizon Annotation DOI: XXXXXXX.XXXXXXX Conference: European Conference on Artificial Intelligence, CLEAR-AI Workshop: Collaborative Methods and Tools for Engineering and Evaluating Transparency in AI; October 25–30, 2025; Bologna, Italy CCS: Computing methodologies Machine learning Teodor Chiaburu, Vipin Singh, Frank Haußer, Felix Bießmann Abstract . Uncertainty quantification is essential in human-machine collaboration, as human agents tend to adjust their decisions based on the confidence of the machine counterpart. Reliably calibrated model uncertainties, hence, enable more effective collaboration, targeted expert intervention and more responsible usage of Machine Learning (ML) systems. Conformal prediction has become a well established model-agnostic framework for uncertainty calibration of ML models, offering statistically valid confidence estimates for both regression and classification tasks. In this work, we apply conformal prediction to SoilNet , a multimodal multitask model for describing soil profiles. We design a simulated human-in-the-loop (HIL) annotation pipeline, where a limited budget for obtaining ground truth annotations from domain experts is available when model uncertainty is high. Our experiments show that conformalizing SoilNet leads to more efficient annotation in regression tasks and comparable performance scores in classification tasks under the same annotation budget when tested against its non-conformal counterpart. All code and experiments can be found in our repository: https://github.com/calgo-lab/BGR . 1. Introduction Monitoring and characterizing soil profiles is essential for agriculture and environmental management. Soil health has a direct impact on each of us, since this is where most of our food is coming from. Accurate soil descriptions support decisions in land use planning and crop suitability. However, traditional soil analysis relies heavily on manual inspection by trained experts, making large-scale monitoring expensive and time-consuming. As a result, automating parts of the soil annotation process has become an important goal, especially in the face of the climate change effects, which highly affects soil quality ( des 2018 ) . Complete soil description is typically a multimodal problem ( Hrsg.: K.J. Hartmann; A. Bauriegel; U. Dehner; E. Eberhardt; S. Hesse; D. Kühn; W. Martin; F. Waldmann; AG Boden 2024 ) that integrates visual input - such as soil profile photographs (see Figure 2 ) - with geotemporal context and chemical/morphological measurements. It also involves a structured series of interdependent tasks: segmenting the soil into meaningful horizons, describing each layer’s morphological properties e.g. humus content, carbonate presence, and finally assigning a horizon class label. Each task depends on the output of the previous one, forming a pipeline where early-stage errors can propagate downstream. In such settings, uncertainty is inevitable, especially given the high complexity of the label taxonomy. To ensure that the desired automated systems can be reliably integrated into expert workflows, it is necessary to design tailored human-AI collaboration strategies that foster transparency and trust. Central to this is the ability of the AI system to effectively communicate its uncertainty to the human stakeholders involved. When uncertainty is well-calibrated and interpretable, experts can make informed decisions about when to rely on the model’s outputs and when to intervene. This not only improves overall system performance but also encourages responsible deployment in high-stakes use cases. In this work, we make the following contributions: (1) We conformalize an existing multimodal, multitask solution for describing soil horizons - SoilNet ( Chiaburu et al. 2025 ) - to provide calibrated uncertainty estimates for two of its three tasks: depth marker prediction and horizon classification. The model’s design already mirrors the workflow of human geologists; by equipping it further with calibrated uncertainties, we aim at rendering the model’s reasoning process even more transparent. (2) We propose an evaluation protocol inspired by a real-world application of the proposed model, assuming a limited annotation budget, which enables systematic evaluation of different uncertainty quantification strategies across both regression and classification tasks. This framework allows us to study how uncertainty-aware collaboration improves annotation outcomes while minimizing expert involvement. 2. Related Work Active Learning. Active Learning (AL) is a paradigm in Machine Learning (ML) where the model selectively queries the most informative data points for annotation, with the goal of improving performance using fewer labeled samples ( Settles 2012 ) . This approach is particularly valuable in domains where labeling is costly or requires expert knowledge. Classical strategies include uncertainty sampling , where the model queries examples on which it is least confident ( Lewis and Gale 1994 ; Liu and Li 2023 ) and query-by-committee , where multiple models "vote" and samples with high disagreement are selected ( Seung et al. 1992 ; Freund et al. 1997 ) . Inspired by the evaluation protocols of active learning strategies we also assume a limited labeling budget. The key difference to our work is that while active learning requires a limited label budget during training, we assume such a budget at inference time to minimize the human expert annotation workload. Quantifying Epistemic Uncertainty. Epistemic uncertainty reflects the model’s lack of knowledge and is particularly important in sensitive applications where incorrect predictions can have serious consequences. To estimate this form of uncertainty, ML systems must move beyond point predictions and instead produce a distribution over possible outputs. Approaches to estimating output distributions of ML models can be broadly categorized in model specific uncertainty estimates and model agnostic approaches. Model specific approaches often use dedicated uncertainty aware models, such as Bayesian Neural Networks (BNNs), often approximated using methods such as Monte Carlo Dropout (MCD) ( Gal and Ghahramani 2016 ; Bykov et al. 2020 ) , Deep Ensembles ( Lakshminarayanan et al. 2017 ) or Variational Inference ( Blundell et al. 2015 ; Graves 2011 ) . Model agnostic approaches instead can be used with arbitrary ML models. A popular type of model agnostic uncertainty estimators is Conformal Prediction ( Vovk et al. 2005 ; Shafer and Vovk 2008 ; Angelopoulos and Bates 2021 ) , which offers distribution-free, finite-sample guarantees on predictive sets or intervals. Conformal methods wrap around any base model and calibrate uncertainty post-hoc using a held-out calibration set, enabling models to produce uncertainty estimates. In this work, we adapt conformal prediction to a human-AI collaborative multi-task annotation pipeline. Multitask Uncertainty and Expert HIL Strategies. Recent work has explored uncertainty-aware models that defer difficult cases to human experts, especially when multiple tasks or modalities are involved. For example, ( Cresswell et al. 2024 ) demonstrate that showing conformal prediction sets significantly improves human–AI team accuracy on classification, sentiment analysis and NER (Named Entity Recognition) tasks, as opposed to simple top- K prediction sets. In their experiments, they use the conformal set size as a measure for the difficulty the model had to solve the task at hand and show that it correlates with human annotation accuracy (the larger the set size, the more the human performance decreases). A related line of work is learning to defer (L2D), where a model is trained to output “not sure – ask an expert” via a deferral loss . In ( Montreuil et al. 2025 ) , the authors develop a two-stage L2D framework for ’multi-task medical diagnosis’ that involve both classification (predicting mortality) and regression (length of hospital stay). In their setting, if the model is sufficiently confident, its diagnosis is accepted; otherwise, the decision is deferred to a medical expert. Similarly, other works use classification

Read the complete text