Towards Human-AI Complementarity in Matching Tasks

Adrian Arnaiz-Rodriguez, Nina Corvelo Benz, Suhas Thejaswi, Nuria Oliver, Manuel Gomez-Rodriguez · 2025

Read Towards Human-AI Complementarity in Matching Tasks by Adrian Arnaiz-Rodriguez, Nina Corvelo Benz, Suhas Thejaswi, Nuria Oliver, Manuel Gomez-Rodriguez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13285.txt.

Introduction

Matching people to services is high-stakes work, and a 2025 paper by Adrian Arnaiz-Rodriguez and co-authors designs a system where human and algorithm each do what they do best. Algorithmic matchers can beat humans on average, yet a person using one does not always end up ahead of either alone.

Their system, called comatch, makes only the matching calls it is most sure of and hands the rest to the human decision maker. It tunes how many calls to take and how many to defer so the joint result is provably as strong as possible. A study with eight hundred people backed the design: the paired outcomes beat both the people working alone and the algorithm working alone.

Catalog record

CREATORAdrian Arnaiz-Rodriguez, Nina Corvelo Benz, Suhas Thejaswi, Nuria Oliver, Manuel Gomez-Rodriguez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13285

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Human-AI Complementarity in Matching Tasks Adrian Arnaiz-Rodriguez, Nina Corvelo Benz, Suhas Thejaswi, Nuria Oliver, Manuel Gomez-Rodriguez Abstract Data-driven algorithmic matching systems promise to help human decision makers make better matching decisions in a wide variety of high-stakes application domains, such as healthcare and social service provision. However, existing systems are not designed to achieve human-AI complementarity: decisions made by a human using an algorithmic matching system are not necessarily better than those made by the human or by the algorithm alone. Our work aims to address this gap. To this end, we propose collaborative matching ( CoMatch ), a data-driven algorithmic matching system that takes a collaborative approach: rather than making all the matching decisions for a matching task like existing systems, it selects only the decisions that it is the most confident in, deferring the rest to the human decision maker. In the process, CoMatch optimizes how many decisions it makes and how many it defers to the human decision maker to provably maximize performance. We conduct a large-scale human subject study with 800 800 participants to validate the proposed approach. The results demonstrate that the matching outcomes produced by CoMatch outperform those generated by either human participants or by algorithmic matching on their own. The data gathered in our human subject study and an implementation of our system are available as open source at https://github.com/Networks-Learning/human-AI-complementarity-matching . 1 Introduction As AI systems become more integrated into high-stakes decision-making processes, understanding how they interact with human users has become a growing concern. There is increasing consensus that the ultimate goal of machine learning systems for decision support is to achieve human-AI complementarity [ 1 , 2 , 3 , 4 , 5 ] . Human-AI complementarity aims for the decisions made by a human using a decision support system to be, in expectation, better than those made by either the human or the AI system alone. In recent years, there have been significant advances in developing decision support systems that achieve human-AI complementarity by design [ 6 , 7 , 8 , 9 ] . However, these advances have predominantly focused on classification tasks. In this work, we focus on decision support systems designed to achieve human-AI complementarity in matching tasks. Matching tasks appear in a variety of high-stakes application domains, including matching refugees to resettlement locations [ 10 , 11 , 12 , 13 , 14 ] , patients to appointments with clinicians [ 15 ] , or blood and organ donors to recipients [ 16 , 17 ] . In these cases, a decision maker needs to distribute a limited set of resources, such as locations, appointments, or donations, among a pool of individuals, such as refugees, patients, or recipients, through matching decisions so that individuals will use the resources effectively. However, at the time of assignment, there is uncertainty as to whether a given individual will actually benefits from the resource. In this context, data-driven algorithmic matching systems leverage machine learning models to predict how effectively each individual under consideration would use a given resource. 1 1 1 In practice, one needs to measure whether an individual makes good use of a given resource using proxy variables which need to be chosen carefully to avoid perpetuating historical biases [ 18 , 19 , 20 ] . Then, they leverage these predictions to make matching decisions by solving a maximum weight bipartite matching problem [ 21 , 22 ] where nodes represent individuals and resources, respectively, and edge weights represent the classifier’s confidence that an individual would effectively use a given resource. The goal is to find an assignment between individuals and resources that maximizes the sum of the edge weights. While such algorithmic matching systems can optimize predicted outcomes, they still rely on human decision makers to achieve effective human-AI complementarity [ 11 ] . In practice, this means decision makers must be able to interpret and, when necessary, override the system’s recommendations. However, there is growing evidence that decision makers often struggle to make these judgments appropriately [ 23 , 24 , 25 , 26 ] , which can undermine the potential benefits of the system. In this work, we propose a data-driven algorithmic matching system designed to achieve human-AI complementarity without requiring to understand when or how to override the system’s recommendations. In detail, our contributions in this work are three-fold: 1. A collaborative algorithmic matching framework: We propose Collaborative Matching ( CoMatch ), a data-driven algorithmic matching framework that, unlike traditional matching frameworks, does not make all matching decisions in a matching task. 2 2 2 Throughout this paper, we refer to a matching task as the process of assigning a subset of individuals to available time slots, and to a matching decision as the assignment of an individual to a specific time slot within that task. Instead, it matches only the individual–resource pairs that it is most certain about—by solving a maximum weight bipartite imperfect matching problem [ 27 ] —and defers the remaining decisions to a human decision maker. Along the way, our framework utilizes UCB1 , a well-known multi-armed bandit algorithm [ 28 ] , to optimize how many matching decisions within each matching task instance to defer to the human decision maker to provably maximize the performance. 2. Empirical validation through a large-scale human subject study: We conduct a large-scale human subject study with 800 participants. They completed 6,400 matching tasks, each of which involving the assignment of a subset of individuals to time slots. This resulted in a total of 80,000 individual matching decisions across 40 distinct task samples. In this setup, human participants have access to more information than the classifier simulating a realistic human-AI collaboration scenario, allowing us to evaluate the benefits of deferring matching decisions to human decision-makers. 3. Open data and implementation: To facilitate future research and ensure reproducibility, we release the full dataset collected from our user study and an implementation of the proposed matching framework as open-source at https://github.com/Networks-Learning/human-AI-complementarity-matching . Further Related Work. Our work builds upon related work on learning under algorithmic triage and multi-armed bandits. The literature on algorithmic triage aims to develop classifiers that make predictions for a given fraction of the samples and leave the remaining to human experts, as instructed by a triage policy [ 29 , 30 , 31 , 32 , 33 , 34 , 35 ] . Here, the triage policy determines who predicts each sample independently of each other. In contrast, the proposed system determines who makes each matching decision for each individual in a pool jointly by solving a linear program. In this context, note that learning under algorithmic triage has been also extended to reinforcement learning settings [ 36 , 37 , 38 , 39 ] . Furthermore, our research contributes to an extensive line of work that uses multi-armed bandits in real-world applications [ 40 , 41 , 42 , 43 , 44 , 45 , 8 ] . Within this area of research, our work is most closely related to that of Straitouri et al. [ 8 ] , which has used multi-armed bandits to optimize the performance of decision support systems for classification tasks based on prediction sets.

Read the complete text