Few-shot Policy (de)composition in Conversational Question Answering

Kyle Erwin, Guy Axelrod, Maria Chang, Achille Fokoue, Maxwell Crouse, Soham Dan, Tian Gao, Rosario Uceda-Sosa, Ndivhuwo Makondo, Naweed Khan, Alexander Gray · 2025

Read Few-shot Policy (de)composition in Conversational Question Answering by Kyle Erwin, Guy Axelrod, Maria Chang, Achille Fokoue, Maxwell Crouse, Soham Dan, Tian Gao, Rosario Uceda-Sosa, Ndivhuwo Makondo, Naweed Khan, Alexander Gray (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11335.txt.

Introduction

Checking whether a story obeys a written policy sounds simple, until the policy hides conditions inside conditions. Kyle Erwin and ten co-authors propose LDPC in 2025, a framework that makes a language model do the checking out loud. It splits policies into sub questions and assigns each a truth value from the conversation. Then it writes the logic down as explicit statements. Only a few examples are shown to the model.

On the standard ShARC benchmark the method matched systems trained for the task, without any task specific training. Because the logic is written out, errors can be traced to ambiguous benchmark items as often as to the model. Transparency, the authors argue, is a feature compliance work cannot skip.

Catalog record

CREATORKyle Erwin, Guy Axelrod, Maria Chang, Achille Fokoue, Maxwell Crouse, Soham Dan, Tian Gao, Rosario Uceda-Sosa, Ndivhuwo Makondo, Naweed Khan, Alexander Gray
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11335

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Few-shot Policy (de)composition in Conversational Question Answering Kyle Erwin, Guy Axelrod, Maria Chang, Achille Fokoue, Maxwell Crouse, Soham Dan, Tian Gao, Rosario Uceda-Sosa, Ndivhuwo Makondo, Naweed Khan, Alexander Gray Abstract The task of policy compliance detection (PCD) is to determine if a scenario is in compliance with respect to a set of written policies. In a conversational setting, the results of PCD can indicate if clarifying questions must be asked to determine compliance status. Existing approaches usually claim to have reasoning capabilities that are latent or require a large amount of annotated data. In this work, we propose logical decomposition for policy compliance (LDPC): a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting. By selecting only a few exemplars alongside recently developed prompting techniques, we demonstrate that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies. The formulation of explicit logic graphs can in turn help answer PCD-related questions with increased transparency and explainability. We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific fine-tuning. We also leverage the inherently interpretable architecture of LDPC to understand where errors occur, revealing ambiguities in the ShARC dataset and highlighting the challenges involved with reasoning for conversational question answering. \DeclareCaptionType Example 1 Introduction Policy compliance detection (PCD) is the task of automatically determining if a scenario complies with or violates a policy. Automated methods for PCD can be used across a variety of domains, e.g. detecting regulatory violations, determining benefits eligibility, or enforcing social media guidelines. In most real-world applications, scenarios are typically underspecified, making it difficult to determine compliance without obtaining additional information. In a conversational setting, additional information can be obtained by generating follow-up questions in a task referred to as conversational machine reading Saeidi et al. 2018 . One way to approach the PCD task is to decompose policies into questions and combine those questions into logical expression trees that can be used to arrive at the final answer Saeidi et al. 2021 . Large language models (LLMs) are good candidates for conversational PCD, given their impressive natural language generation abilities that resemble deep understanding and emergent reasoning. However, LLMs have been shown to have issues with factuality Ji et al. 2023 and reasoning Valmeekam et al. 2022 . Many approaches seek to improve factuality and reasoning capabilities by having LLMs “show their work” Wei et al. 2022 ; Yao et al. 2024 , though even these elicited “thoughts” may not accurately reflect the factors that influence LLMs outputs Turpin et al. 2024 . To mitigate the risks of incorrect or unfaithful explanations, a variety of self-improving LLMs have been proposed Wang et al. 2022 ; Pan et al. 2023 as a way to decompose the opaque nature of LLM inference. In this paper, we propose Logical Decomposition for Policy Compliance (LDPC), which uses LLM-based reasoning that can be applied to the PCD task. Specifically, our system decomposes a natural language policy into yes/no questions that can be composed into an explicit logical formula to be used for PCD and for asking follow up questions. Importantly, the PCD decision is deterministically inferred from the logical formula, as opposed to being generated in an opaque manner and tenuously connected to an explanation or chain of thought. We make the following contributions in this paper: 1. We present Logical Decomposition for Policy Compliance (LDPC): an inherently interpretable approach to conversational PCD that combines neural question decomposition and logic formulation with symbolic reasoning. 2. Our policy decomposition and logical formulation modules receive no task-specific fine-tuning, yet achieve competitive results on the ShARC dataset. 3. We propose a simple but powerful three-valued logic that is especially suitable for conversational PCD. 4. We identify ambiguities in the ShARC dataset and challenges that arise from reasoning about policies to answer questions in a conversational setting 2 Related Work This paper builds upon prior work in two general areas: conversational machine reading (Section 2.1 ) and reasoning with large langauge models via prompt-based instruction (Section 2.2 ). 2.1 Conversational Machine Reading Conversational machine reading is a type of question answering task that requires the joint interpretation of a scenario and some background information to answer a question, where the appropriate response to the question may not be an answer at all, but rather, a follow-up question Saeidi et al. 2018 . It was introduced as a more challenging and realistic alternative to question answering tasks where answers were spans of text that could be extracted from a paragraph, as in the highly influential Stanford Question Answering Dataset Rajpurkar et al. 2016 . We leverage two influential datasets in this domain: ShARC Saeidi et al. 2018 and QA4PC Saeidi et al. 2021 . 2.1.1 ShARC The ShARC dataset Saeidi et al. 2018 is a dialogue-based question-answering dataset constructed from 948 distinct text snippets. These snippets are taken from government websites and describe a policy. Each snippet features an input question and a corresponding “dialog tree.” These trees branch based on yes/no answers to follow-up questions at each step. The dataset comprises all individual branching paths, referred to as “utterances”, from these trees. There are 6,058 utterances in total. Additionally, 6,637 scenarios provide extra information about the user, enabling the skipping of certain questions. Scenarios modify dialog trees, resulting in 37,087 distinct utterances when combined with negative sampled scenarios. After removing unreachable portions of dialog trees, the final dataset consists of 32,436 utterances. For each utterance, a model must predict one of the following classes: irrelevant, yes, no, follow-up question. If the model decides that a follow up question is the most appropriate response, it must also generate that follow up question. The dataset is divided into train, development, and test sets, with sizes of 21,890, 2,270, and 8,276, respectively. Evaluation metrics include Micro and Macro Accuracy and BLEU. Micro and Macro Accuracy assess the model’s ability to classify each utterance, while BLEU evaluates the quality of generated questions. Many techniques have been introduced to compete for top performance on ShARC. For example, Zhong and Zettlemoyer 2019 train a model to extract rules from policy text and determine which rules are entailed by the conversation history. Using self-attention layers for the input and extracted rules, a decision module is able to learn how to rephrase rules into follow up questions and when to ask those questions. In contrast, Lawrence et al. 2019 use bidirectional attention to make predictions on ShARC. The dialogue graph modeling (DGM) algorithm ( Ouyang et al. 2020 ) improves on Discern Gao et al. 2020 by using explicit discourse graph and implicit discourse graph models with graph convolution network (GCN). ET5 ( Zhang et al. 2022 ) proposes a end-to-end framework with T5 to fully exploit the entailment reasoning in decision making steps by using a shared encoder (with a multi-task architecture) and separate decoders for reasoning and answer generation. T-reasoner ( Sun et al. 2022 ) integrates a reasoning module to model condition relationships and verify consistent answers within user scenarios, identifying unsatisfied

Read the complete text