Benchmarking and Advancing Large Language Models for Local Life Services Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.

Xiaochong Lan, Jie Feng*, Jiahuan Lei, Xinlei Shi, Yong Li* · 2025

Read Benchmarking and Advancing Large Language Models for Local Life Services Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. by Xiaochong Lan, Jie Feng*, Jiahuan Lei, Xinlei Shi, Yong Li* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02720.txt.

Introduction

Xiaochong Lan and four co-authors test, in a 2025 study, how well large language models handle local life services. Such services cover food delivery, bookings, and errands near a user, a field tied to the Meituan platform. The team built a comprehensive benchmark for the field and ran many models through tasks drawn from it.

Two improvement paths were then tried: fine-tuning on field data, and agent-based workflows that break tasks into steps. A compact model with seven billion parameters, once tuned, matched a much larger model with seventy-two billion. That balance of cost and capability matters for live services, where each call must be fast and cheap. The authors say tuned small models make such AI help practical for daily local applications. The work also sets a shared yardstick for the field.

Catalog record

CREATORXiaochong Lan, Jie Feng*, Jiahuan Lei, Xinlei Shi, Yong Li*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02720

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking and Advancing Large Language Models for Local Life Services Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25), August 3–7, 2025, Toronto, ON, Canada DOI: 10.1145/3711896.3737196 ISBN: 979-8-4007-1454-2/2025/08 CCS: Computing methodologies Artificial intelligence CCS: Information systems Information systems applications Xiaochong Lan, Jie Feng*, Jiahuan Lei, Xinlei Shi, Yong Li* Abstract . Large language models (LLMs) have exhibited remarkable capabilities and achieved significant breakthroughs across various domains, leading to their widespread adoption in recent years. Building on this progress, we investigate their potential in the realm of local life services. In this study, we establish a comprehensive benchmark and systematically evaluate the performance of diverse LLMs across a wide range of tasks relevant to local life services. To further enhance their effectiveness, we explore two key approaches: model fine-tuning and agent-based workflows. Our findings reveal that even a relatively compact 7B model can attain performance levels comparable to a much larger 72B model, effectively balancing inference cost and model capability. This optimization greatly enhances the feasibility and efficiency of deploying LLMs in real-world online services, making them more practical and accessible for local life applications. Available resources are at https://github.com/tsinghua-fib-lab/LocalEval . † † cc-license: by 1 1 footnotetext: Corresponding author 1. Introduction Large language models (LLMs) have achieved remarkable success across various domains, from conversational AI ( OpenAI 2022 ) to mathematical problem solving ( Azerbayev et al. 2023 ; Trinh et al. 2024 ) . However, despite their impressive capabilities on general tasks, LLMs often struggle with domain-specific applications where specialized knowledge is crucial ( Wei et al. 2022 ; OpenAI 2022 ; Guo et al. 2025 ; Zheng et al. [n. d.] ) . This limitation has driven the development of domain-specific LLMs tailored for particular fields, such as medical diagnosis ( Singhal et al. 2023 ) , financial analysis ( Wu et al. 2023 ) , and code generation ( Hui et al. 2024 ; Zhu et al. 2024 ) . Local life services represent a massive yet underexplored opportunity for LLM applications. These services fulfill users’ daily needs through location-based offerings, such as dining, accommodation, personal care, education, and entertainment, among others. Unlike traditional e-commerce, local life services are characterized by their inherent complexity: they involve physical interactions, temporal constraints, and localized preferences that vary across regions and cultures ( Lan et al. 2023 ; Liu et al. 2022 ; Chen et al. 2022 ; Liu et al. 2025 ) . Consider how a user’s restaurant choice depends not only on cuisine preferences but also on real-time factors like time, weather, traffic conditions, special occasions, and even subtle cultural contexts that change from neighborhood to neighborhood. This intricate interplay of factors creates fundamental challenges for computational understanding and decision-making in local life service platforms. Meituan, serving as China’s largest local life service platform, exemplifies both the scale of this opportunity and the diversity of challenges involved. The platform requires sophisticated understanding across numerous tasks: interpreting merchant information and service capabilities, analyzing user preferences and behavioral patterns, predicting consumption trends under various spatiotemporal contexts, generating personalized content, and facilitating effective user-merchant interactions, among others ( Lan et al. 2023 ; Ping et al. 2021 ) . Each task demands not only statistical learning from historical data but also common-sense reasoning about human behavior, cultural knowledge, and contextual understanding. Traditional machine learning approaches often fall short, particularly for long-tail scenarios where data sparsity makes pure statistical learning ineffective. LLMs, with their broad knowledge base and reasoning capabilities, offer a promising solution. However, two critical gaps prevent their effective deployment. First, no systematic evaluation framework exists to assess LLM performance across the diverse tasks required in local life service platforms. Second, the domain shift between web-centric training data and the offline, location-specific nature of local services severely limits general-purpose LLMs, while the scarcity of domain-specific instruction data hinders the development of specialized models. In this paper, we propose a framework to evaluate and enhance LLMs for local life service platforms. Our approach addresses both challenges through systematic evaluation and targeted model improvement. First, we introduce LocalEval, a comprehensive benchmark comprising over 40 tasks organized into four categories: service fundamentals, spatiotemporal context, user interaction, and composite tasks. This benchmark enables rigorous assessment of both general-purpose and domain-specific LLMs. Second, we develop a multi-agent instruction synthesis method that transforms raw platform data into high-quality training examples, enabling smaller models to achieve performance comparable to much larger ones. Third, we design agentic workflows for complex composite tasks, incorporating expert knowledge to guide model reasoning and generate additional training data. Our experiments demonstrate that through targeted instruction tuning, a 7B parameter model can match the performance of 72B models on local life service tasks, significantly reducing deployment costs while maintaining effectiveness. Furthermore, real-world deployments in Meituan’s recommendation, search, and review ranking systems validate the practical impact of our approach. In summary, our contributions are fourfold. • To the best of our knowledge, we are the first to propose a systematic framework for evaluating and applying LLMs in local life service platforms. • We build a comprehensive offline benchmark to assess the performance of various open-source and proprietary LLMs for local life services. • We propose a multi-agent-based method for domain-specific instruction data synthesis, enabling the successful training of small LLMs to achieve competitive performance compared with large-scale LLMs, with minimal computing overhead. • Extensive experiments conducted on both offline benchmark and online scenarios demonstrate the effectiveness of the proposed framework and the fine-tuned domain-enhanced LLMs. 2. Method Our approach consists of three main parts, as illustrated in Figure 1 . First, to systematically evaluate existing LLMs’ capabilities in local life service tasks, we develop LocalEval, a benchmark consisting of various tasks about local life services. Second, we design an instruction tuning approach to enhance LLMs’ understanding of local life services. For composite tasks within the benchmark, we develop agentic workflows to better address these challenges. In the following sections, we detail each part of our approach. Figure 1 . An overview of our approach. We first develop LocalEval Benchmark to systematically evaluate LLMs’ capabilities in understanding local life services. Based on a multi-agent collaboration approach, we construct LocalInstruction to enhance LLMs’ service understanding capabilities through fine-tuning. For composite tasks, we implement expert agents to further improve LLMs’ problem-solving abilities. 2.1. Benchmark Large language models have shown rapid development in recent years with significant progress across various domains. With their understanding of daily life knowledge, LLMs have potential in addressing local life service-related tasks on life service platforms. However, there has been no systematic evaluation of LLM capabilities in the local life services domain. We introduce LocalEval, a benchmark designed to assess LLM capabilities in local life services and establish metrics for future iterations. On local life service platforms, all events can be understood as interactions between users and services in specific spatiotemporal contexts. Following this perspective, we design our benchmark with four task categories: Service Fundamentals, Service with (Spatiotemporal) Context, User-Service Interaction, and Composite Tasks. 2.1.1. Service Fundamentals. The Meituan platform offers diverse local life

Read the complete text