SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation

Shrikant Kendre [email protected], Austin Xu [email protected], Honglu Zhou [email protected], Michael S. Ryoo [email protected], Shafiq Joty [email protected], Juan Carlos Niebles [email protected] · 2025

Read SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation by Shrikant Kendre [email protected], Austin Xu [email protected], Honglu Zhou [email protected], Michael S. Ryoo [email protected], Shafiq Joty [email protected], Juan Carlos Niebles [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17432.txt.

Introduction

Shrikant Kendre and five co-authors introduce SMILE, a metric for grading question-answering systems that tries not to be fooled by wording. Classic metrics score answers by overlapping word sequences, missing meaning; embedding-based metrics capture meaning but drop exact wording and cannot balance sentence-level against keyword-level agreement. Grading by a large language model works but is costly and inconsistent. SMILE combines the three ingredients: sentence-level semantics, keyword-level semantics, and easy keyword matching.

The composite balances lexical precision against semantic relevance, staying computationally light. Across text, image, and video question-answering benchmarks, SMILE's scores correlated strongly with human judgments. The authors present it as a practical middle path for evaluation. Benchmarks need to reward answers that are both correctly worded and genuinely right, without renting a frontier model to judge every response.

Catalog record

CREATORShrikant Kendre [email protected], Austin Xu [email protected], Honglu Zhou [email protected], Michael S. Ryoo [email protected], Shafiq Joty [email protected], Juan Carlos Niebles [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17432

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation Shrikant Kendre [email protected], Austin Xu [email protected], Honglu Zhou [email protected], Michael S. Ryoo [email protected], Shafiq Joty [email protected], Juan Carlos Niebles [email protected] Abstract Traditional evaluation metrics for textual and visual question answering, like ROUGE, METEOR, and Exact Match (EM), focus heavily on n-gram based lexical similarity, often missing the deeper semantic understanding needed for accurate assessment. While measures like BERTScore and MoverScore leverage contextual embeddings to address this limitation, they lack flexibility in balancing sentence-level and keyword-level semantics and ignore lexical similarity, which remains important. Large Language Model (LLM) based evaluators, though powerful, come with drawbacks like high costs, bias, inconsistency, and hallucinations. To address these issues, we introduce SMILE : Semantic Metric Integrating Lexical Exactness, a novel approach that combines sentence-level semantic understanding with keyword-level semantic understanding and easy keyword matching. This composite method balances lexical precision and semantic relevance, offering a comprehensive evaluation. Extensive benchmarks across text, image, and video QA tasks show SMILE is highly correlated with human judgments and computationally lightweight, bridging the gap between lexical and semantic evaluation. Code and evaluation scripts are available at https://github.com/SalesforceAIResearch/smile-metric-qna-eval 1 Introduction Question answering (QA) is an essential task used to measure the progress of language-based models. Across text, image, and video domains, the primary measure of model performance on QA benchmarks is accuracy, which is typically computed via exact (or easy) match (EM) : A model response is deemed correct if the ground-truth answer, typically annotated by humans, exactly matches (or can be found within) the model response. As recent models have grown to be more capable language generators, model answers have grown more nuanced, making EM overly stringent ( 47 ) . A reasonable attempt to mitigate these issues is to employ N-gram based metrics typically used for text generation evaluation, such as ROUGE ( 27 ) or METEOR ( 5 ) , or embedding-based metrics like BERTScore ( 60 ) , MoverScore ( 61 ) , and BLEURT ( 42 ) , to assess similarity between the predicted response and the ground-truth ( 39 ; 4 ; 10 ; 20 ; 57 ) . While such metrics capture high-level similarity between the model response and ground-truth, they may miss fine-grained details crucial to answer correctness (e.g., “The cat is on a chair” vs. “The cat is under a chair”) that result in lower correlation with human judgments ( 33 ) . Concurrently, due to their strong language comprehension abilities, large language models (LLMs) have been deployed as automatic evaluators for text generation. This approach, broadly known as LLM-as-judge , functions by either prompting more capable LLMs, like GPT-4o, or finetuning smaller LLMs specifically for evaluation. LLM-as-judge is appealing as LLMs can adapt to different evaluation criteria and generate explanations. Consequently, recent methods and benchmarks ( 17 ; 49 ) now employ judges as evaluators in QA settings. However, using judge models for evaluation increases costs . For practitioners and developers with limited resources, repeatedly querying pay-per-use API models to evaluate large datasets (5K+ samples) or dedicating limited compute to hosting an evaluation server can be impractical for rapid development, which may drive them to use lesser but faster metrics. Beyond resource demands, generative evaluators also exhibit relatively high latency (see Figure 1 ) and are susceptible to hallucinations, as we qualitatively show in Section 3 . Figure 1 : SMILE offers high-performance, low-latency QA evaluation , breaking the trade-off between cost and performance. Performance is averaged across human-annotated samples from benchmarks for natural language ( \times ), image ( \triangle ), and video ( \circ ) domains. This work revisits embedding-based approaches for automatic QA evaluation and introduces Semantic Metric Integrating Lexical Exactness (SMILE), a lightweight yet high-performing framework for grading QA tasks. SMILE aims to retain the efficiency of embedding-based evaluators while addressing their limitations, such as lack of fine-grained response understanding. To do so, SMILE comprises two subscores: A semantic subscore to assess overall response content, and a keyword subscore to reward lexical exactness. Overall, SMILE offers a best-of-both-worlds evaluation solution: As Figure 1 shows, it correlates with human annotators as strongly as GPT-4o. Additionally, SMILE core components can be precomputed for fast lookup, resulting in a 9x speedup compared to API queries. SMILE’s lightweight design allows it to run on CPU during evaluation, requiring minimal GPU VRAM to perform a one time evaluation dataset preprocessing step. Additionally, SMILE offers interpretability through its composite structure with two subscores. While SMILE can be applied in many settings, our study focuses on factoid QA tasks. Our contributions are: (1) We revisit the promise of embedding-based automatic evaluation metrics for QA tasks and propose SMILE, which utilizes both sentence and keyword level similarity scores to evaluate based on holistic and fine-grained content. (2) We construct a 225-sample human annotated test set from nine (text, image, video) QA datasets, labeled by 4 domain experts. This set is used to benchmark SMILE and other metrics based on their correlation with human judgments. (3) We demonstrate that SMILE can serve as a lightweight drop-in replacement for more powerful LLM-as-judge models across modalities. (4) Extensive ablation studies demonstrate the necessity of each of SMILE’s components. In all, our experiments demonstrate that SMILE is a lightweight yet high-performing automatic evaluation metric for QA settings. Code and evaluation scripts are available at https://github.com/SalesforceAIResearch/smile-metric-qna-eval 2 Background and related work This work addresses three QA modalities: Natural language QA (NLQA), visual QA (VQA), and video QA (VidQA). Across these, a model f receives an input (q,c) and generates a textual answer y=f(q;c) . The input consists of a question q and context c , where c is text for NLQA, an image for VQA, and a video for VidQA. The task of the model is to produce a natural language answer y to the question q based on the given context c . For model evaluation, we adopt a reference-based protocol, assuming a human-annotated ground-truth answer y^{\star} is available for each input (q,c) . Given a model response y , the goal is to determine its correctness. Specifically, we aim to design an evaluator j that produces an evaluation score s=j(y,y^{\star}) based on y and y^{\star} . This source-free setup, where evaluation occurs without access to (q,c) , aligns with the original exact match (EM) evaluation setup. Source-free evaluation may represent an easier evaluation setting, as prior work indicates LLM-based evaluators struggle when context c is included ( 53 ) . For practitioners prioritizing accuracy, the evaluation score s can be a binary correct/incorrect label. For detailed failure analysis, a finer-grained score (e.g. 0-5 scale) may be preferred. Regardless of the format, the score should be convertible to a binary label, typically via a straightforward threshold (e.g., scores \leq 3 as incorrect, scores \geq 4 as correct for a 0-5 scale) ( 32 ; 49 ) . We now review existing metrics and evaluators. Text generation metrics. QA benchmarks have typically used EM or ROUGE ( 27 ) to assess model outputs, e.g., ( 58 ; 39 ; 10 ; 11 ) . As model responses grew more nuanced, n-gram metrics such as BLEU ( 37 ) and METEOR ( 5 ) were adopted in QA settings ( 4

Read the complete text