A Comprehensive Survey on the Risks and Limitations of Concept-based Models

Sanchit Sinha, Aidong Zhang · 2025

Read A Comprehensive Survey on the Risks and Limitations of Concept-based Models by Sanchit Sinha, Aidong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04237.txt.

Introduction

Sanchit Sinha and Aidong Zhang survey in 2025 the risks and limits of concept based models, a class of explainable networks that reason through human named ideas. Such models are prized in medicine and finance because each call comes with a reason. Their spread into tender fields has raised the stakes for trust.

Recent work found cracks: ideas leak into each other, inner codes tangle, and small tweaks can flip calls. Whether human fixes during use truly help is also open. The survey gathers these faults and the design choices that blunt them. Both teacher fed and self taught concept models are covered.

Catalog record

CREATORSanchit Sinha, Aidong Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-04237

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Comprehensive Survey on the Risks and Limitations of Concept-based Models Sanchit Sinha, Aidong Zhang Abstract Concept-based Models are a class of inherently explainable networks that improve upon standard Deep Neural Networks by providing a rationale behind their predictions using human-understandable ‘concepts’. With these models being highly successful in critical applications like medical diagnosis and financial risk prediction, there is a natural push toward their wider adoption in sensitive domains to instill greater trust among diverse stakeholders. However, recent research has uncovered significant limitations in the structure of such networks, their training procedure, underlying assumptions, and their susceptibility to adversarial vulnerabilities. In particular, issues such as concept leakage, entangled representations, and limited robustness to perturbations pose challenges to their reliability and generalization. Additionally, the effectiveness of human interventions in these models remains an open question, raising concerns about their real-world applicability. In this paper, we provide a comprehensive survey on the risks and limitations associated with Concept-based Models. In particular, we focus on aggregating commonly encountered challenges and the architecture choices mitigating these challenges for Supervised and Unsupervised paradigms. We also examine recent advances in improving their reliability and discuss open problems and promising avenues of future research in this domain. 1 Introduction With the rising widespread adoption of Deep Neural Networks (DNNs) in human endeavors, there is a strong demand for transparency and rationales behind their decisions. This has resulted in significant research on the explainability of trained models as well as designing inherently explainable models for sensitive applications in medicine and finance. Early explainability research focused on providing feature attributions Sundararajan et al. 2017 ; Ribeiro et al. 2016 or sample importance Koh and Liang 2017 - providing explanations local to specific features and data samples. However, for a more global explanation, explainability using ‘concepts’ is overwhelmingly preferred. Concepts are abstract human-understandable entities, shared among multiple data samples and are directly relevant to task prediction. Multiple research works have attempted to discover concepts in trained models in a post-hoc manner Kim et al. 2018 ; Ghorbani et al. 2019 with various levels of success. These methods usually employ a set of pre-curated representative concepts as probes to query the models for each sample to be explained. More recently, inherently explainable concept-based models have been at the forefront of explainability research. The process of utilizing concepts as intermediaries in task predictions is commonly called as concept learning . Initial research lines of work have attempted to design architectures that assign importance scores to concepts defined in the input space such as Ghorbani et al. 2019 ; Chen et al. 2019 and utilize them during predictions. However, such models suffer from sampling bias in the training and are limited to visually grounded, non-semantic concepts. More recent research attempts to learn concepts as an intermediate to task prediction, with concepts directly responsible for task prediction. In this paper, we focus on concepts defined in the representation space of models and not in the input space. Concept learning is broadly divided into categories of supervised -i.e. supervised with human-annotated concepts such as Koh et al. 2020 ; Espinosa Zarlenga et al. 2024 or unsupervised -i.e. learned automatically as soft signals from task prediction Alvarez-Melis and Jaakkola 2018 (For a more detailed discussion refer Section 2 ). In this paper, we provide a comprehensive survey on the common risks and limitations faced during designing and deployment of concept-based models. We thoroughly explore and document common vulnerabilities in both supervised and unsupervised concept-based models. Even though a few surveys exist in the field of explainability, extremely few of them focus explicitly on concept-based explainability which is an intense and fast-growing field of research. To our knowledge, only the surveys by Poeta et al. 2023 and Shams Khoozani et al. 2024 focus on concept-based explainability but list only the types of architectures and do not document their vulnerabilities in detail. Elsewhere the survey by Nannini et al. 2025 focuses on the risks of Explainable AI but does not focus on Concept-based Models. We believe that a thorough analysis and discussion on the topic of risks and limitations encountered in concept-based models is highly beneficial in helping stakeholders design and deploy them in practice. In particular, our contributions in this paper can be summarized as: • We provide a comprehensive survey on the risks, limitations and vulnerabilities of Concept-Based Models, across both supervised and unsupervised paradigms. • We also thoroughly document mitigation strategies for common vulnerabilities like concept leakage, lack of concept independence, adversarial vulnerabilities, intervention difficulties, etc. • We provide guidelines for open research problems, and propose future directions to enhance the interpretability, robustness, and reliability of concept-based models in real-world applications. 2 Paradigms of Concept Learning Inherently explainable concept-based models can be broadly divided into two distinct paradigms based on the availability of concept-level human annotations. Supervised Concept Learning: The first paradigm treats the concept learning and task learning as independent problems with the concepts acting as a precursor to task prediction. As a result, it requires a set of human-annotated concept labels in addition to the task labels and treats concepts as signals from concept learning. The concept labels are utilized as supervision for intermediate outputs during model training in addition to task labels. The requirement of concept-level supervision makes this approach completely supervised in the concept space. The seminal work in this field is Concept Bottleneck Models Koh et al. 2020 , which demonstrated for the first time that any DNN-based classification backbone can be converted into an equivalent inherently explainable concept-based model. In addition, the model supports intervention . i.e. ‘correcting’ a model’s wrong prediction by intervening manually in the concept space. As human-annotated concept labels completely supervise the concept space, there is significant trust in model predictions. However, these approaches offer limited scalability due to the enormous human cost of manually annotating each concept label. Some newer approaches have utilized LLMs as knowledge banks to annotate concepts, reducing the human effort in labeling. We discuss them in Section 6 as Large Language Models (LLMs) reduce human effort in labeling but still lie in the broad umbrella of supervised concept learning frameworks. Unupervised Concept Learning: The second paradigm attempts to learn concepts directly from the task itself without the presence of any concept labels. This approach treats concepts as model representations during model learning rather than explicit signals. The task labels act as weak supervision signals to guide concept learning and the lack of concept supervision makes this approach completely unsupervised in the concept space. As opposed to supervised approaches, these approaches do not provide explicit concept interpretation (like labels), which are left to stakeholders to figure out. The seminal work in this field is Self-Explaining Neural Network Alvarez-Melis and Jaakkola 2018 which proposes a Siamese network architecture, wherein the first network extracts representative concepts while the second network learns relevance scores associated with the concepts. The final task prediction is performed as a weighted combination of the concepts and the associated scores. These approaches are seamlessly scalable, requiring no human effort in supervision. On the other hand, as there is no control over the learned concepts, these approaches are by definition low-trust and less reliable overall. A visual summary of

Read the complete text