A systematic mapping study method for software architecture research with LLM-assisted quality assessment Authors: Andrei Olar, Oskar Picus, Laura Cernău, Laura Dioșan, Simona Motogna Published: 2026 ABSTRACT This article describes the protocol used to conduct a systematic mapping study (SMS) of software architecture. The presence of architectural and design instruction in primary studies cannot be determined through search and superficial review alone. A principled quality assessment must be conducted to identify studies that are both relevant and of high quality. LLM assistance offers practical means of performing such assessments in a timely and efficient manner. Therefore, our contribution is threefold:• Guidance for the following SMS steps: tooling setup, database search, deduplication and snowballing, primary study selection, quality assessment, and data extraction.• The rationale and methodology to perform quality assessment on primary studies regarding software architecture.• A method of using large language models (LLMs) together with human evaluators is presented.The method is supported by bespoke open-source software. Specifications table Subject areaComputer ScienceMore specific subject areaSoftware Engineering; Information SystemsName of your methodLLM-assisted systematic mapping study protocolName and reference of original methodPetersen et al., Systematic mapping studies in software engineering, in: 12th International Conference on Evaluation and Assessment in Software Engineering, vol. 17, 2008;Resource availabilityMapwiseFox (open-source): https://github.com/koosie0507/mapwisefox Subject areaComputer ScienceMore specific subject areaSoftware Engineering; Information SystemsName of your methodLLM-assisted systematic mapping study protocolName and reference of original methodPetersen et al., Systematic mapping studies in software engineering, in: 12th International Conference on Evaluation and Assessment in Software Engineering, vol. 17, 2008;Resource availabilityMapwiseFox (open-source): https://github.com/koosie0507/mapwisefox Background Systematic mapping studies (SMS) classify and count primary studies to characterize a research area [1,2]. They differ from systematic literature reviews (SLR), which aggregate and synthesize empirical evidence [3]. The SMS methodology is appropriate when the goal is to describe the distribution of research across categories rather than to answer a specific empirical question. This article describes an SMS protocol that was developed originally to map how entity resolution (ER) systems are architected across the published literature. ER systems identify records that refer to the same real-world entity across one or more data sources. The literature on ER is large and spans multiple communities – database systems, knowledge graphs, natural language processing, and data integration – each using different terminology and producing different types of software artifacts [[4], [5], [6], [7], [8], [9]]. Standard SMS guidance does not require quality assessment. However, even a brief scoping exercise reveals that keyword-based inclusion criteria alone cannot reliably identify studies that provide actionable architectural knowledge. A paper may describe an end-to-end ER process, pass every keyword-based criterion, and still provide insufficient information to envisage how the system is structured, deployed, or extended. Quality assessment was therefore added as an exclusion gate. A second motivation for this protocol is scale. The candidate set produced through automated search and snowballing is large. Assigning every study to two independent human reviewers for quality assessment is feasible but time-consuming. Large language models (LLMs) are used increasingly routinely as judges, with guidelines appearing in non-peer reviewed venues (so far) [10]. Our method for conducting SMS partitions the candidate-set and assigns one portion to LLMs during quality assessment, reducing the human workload but keeping the results auditable. This article provides an LLM assessment component which is validated by comparing LLM scores with human scores on the same articles using four agreement metrics. The methodological contribution of this protocol is therefore not any single component in isolation – systematic mapping, quality assessment, and LLM-based judging each have prior art – but their integration into a single workflow in which quality assessment acts as a mandatory exclusion gate and LLM assistance is used as a complement to human evaluation. In the pilot SMS that validates this protocol [11], all computational steps were carried out with MapwiseFox [12], an open source Python tool built for this very purpose and archived under a persistent identifier for full reproducibility. Method details The present study extends the systematic mapping study (SMS) methodology [1,2]. SLRs aggregate and synthesize empirical evidence to answer specific research questions using detailed quality assessment and often include the meta-analysis of quantitative results. On the other hand, an SMS builds a classification scheme and structures a research area by counting and categorizing primary studies [1]. The analysis focuses on publication frequencies across categories rather than on synthesizing empirical findings highlighting that “Systematic maps and reviews are different in terms of goals, breadth, validity issues and implications” and “should be used complementarily.” [2] SMS methodology was considered a suitable base because the goal is to characterize how ER systems are architected across the literature, not to synthesize empirical evidence about which architectural approach performs best. Using this methodology, one can formulate research questions that ask about publication distributions, terminology, architecture styles, and representation methods. These questions are answered by classification and counting, an approach which aligns with the SMS methodology. Tooling. The computational steps of this protocol – database search, deduplication, snowballing, selection, quality assessment, and workload partitioning – are not tied to any specific tool. Any tool chain that implements the tool-independent specifications in Table 1 satisfies the protocol, as do equivalent manual procedures. As one such implementation, all steps in the protocol may be carried out using MapwiseFox [12], an open-source Python toolkit built for this purpose. It supports searching against citation databases and digital libraries by either running the search automatically against those databases which provide public APIs, or by providing the native query that the user can input in the ‘advanced’ search UI of the databases that have one and publish a query domain-specific language to use it.Table 1Protocol steps and their tool-independent specification with the corresponding MapwiseFox command.Table 1 dummy alt textProtocol StepTool-independent SpecificationMapwiseFox CommandDatabase SearchRun query against each API or web UIsearchDeduplicationDeduplication/entity resolution librarydeduplicateSnowballingQuery citation indexes (e.g. Semantic Scholar, OpenAlex, Crossref) for cited/citing workssnowballSelectionForm/workbook designed to collect include/exclude verdictswebQuality assessmentLLM API call with standardized promptassistantWorkload partitioningRound-robin assignment scriptsplit-workload Table 1Protocol steps and their tool-independent specification with the corresponding MapwiseFox command.Table 1 dummy alt textProtocol StepTool-independent SpecificationMapwiseFox CommandDatabase SearchRun query against each API or web UIsearchDeduplicationDeduplication/entity resolution librarydeduplicateSnowballingQuery citation indexes (e.g. Semantic Scholar, OpenAlex, Crossref) for cited/citing workssnowballSelectionForm/workbook designed to collect include/exclude verdictswebQuality assessmentLLM API call with standardized promptassistantWorkload partitioningRound-robin assignment scriptsplit-workload Protocol steps and their tool-independent specification with the corresponding MapwiseFox command. Table 1 maps this protocol’s steps to tool-independent specifications and the corresponding MapwiseFox command. The tool does not support activities that require direct human collaboration and coordination, such as the reconciliation of inter-rater disagreements in data extraction. Data. All artifacts produced using this protocol must be openly archived on a platform that promotes reproducibility by assigning persistent unique identifiers such as the Open Science Framework (OSF)1 or Zenodo.2 At a minimum, the reproduction package must contain instructions about its contents and the steps to reproduce the systematic mapping study. Search. Preliminary scoping is strongly suggested both by the SMS methodology and this extension. When conducting the related SMS using this method, the appropriate sources and search terms were determined during preliminary scoping. The search databases are usually chosen through manual selection. Guidance for making a more informed choice exists [13]. In the case of the related SMS, the selected citation databases were Scopus and the Web of Science (WoS) Core Collection. Those databases were supplemented using the following digital libraries: ACM Digital Library, ScienceDirect, Springer Link, and IEEE Xplore. This particular selection provides broad coverage of peer reviewed studies [13]. Other citation databases (Semantic Scholar, Crossref and Google Scholar) were considered for the search phase, but even though they also provide ample coverage, they were ultimately excluded because they index non-peer-reviewed material leading to significant noise during the selection process. The second role of preliminary scoping is to identify good search terms. This SMS protocol suggests drawing candidate terms from the titles, abstracts, and keywords of known-good primary studies and from the secondary literature on the problem subjected to mapping. This is commonly followed by running multiple candidate-sets of search terms until finding a stable configuration. Using a tool that ensures that the same query is run against multiple databases is strongly suggested. Terms must be retained when they maximize recall without introducing unacceptable noise. For example, each candidate-set can be evaluated against the selected databases. For entity resolution, this exercise would converge on seven terms that are used predominantly across ER literature: entity resolution, entity matching, record linkage, de-duplication, merge/purge, entity linking, and entity alignment [[4], [5], [6], [7], [8], [9]]. These terms span the principal communities that produce ER systems and cover the most common synonyms identified in the secondary literature on entity resolution. The search should be restricted to primary studies that describe software artifacts. Five search terms served for this purpose: system, architect*, tool*, framework, and library. These terms, used in conjunction with the entity resolution terms, substantially reduce the volume of purely algorithmic or experimental results without eliminating system-oriented studies. The result in our own mapping study was the following final search query: (‘entity resolution’ OR ‘entity matching’ OR ‘record linkage’ OR ‘data deduplication’ OR ‘merge/purge’ OR ‘entity linking’ OR ‘entity alignment’) AND (‘system’ OR ‘architect’ OR ‘tool’ OR ‘framework’ OR ‘library’) The full query was run against the title and the abstract. The keywords of each candidate study were matched only against the part of the query related to entity resolution. Generic terms such as system and framework are too common to serve a useful purpose as keyword filters. The search was further restricted to journal articles and full papers published in conference proceedings. Short papers, extended abstracts, and all categories of secondary study were excluded at the search stage. Where the source allowed, results were further restricted to the computer science subject area. This reduced contamination from medical and epidemiological uses of record linkage, where the term denotes a data-collection methodology rather than a software task, and from natural-language-processing uses of entity linking, where this term is often confused with named entity recognition. The above search query was deliberately written in a citation database agnostic manner. The recommendation in this protocol is to generate all source specific searches automatically from the same query. In the pilot SMS, this recommendation was handled using MapwiseFox [12]. Some sources did not require user involvement beyond the automated tool because they allowed running the queries via a public API. Others (e.g., ACM Digital Library) required user interaction with a web interface. In particular for Springer Link, the API response is broader than the query warrants, so the retrieved records had to be filtered locally by title, abstract, language, and subject classification before being passed to the deduplication step. Table 2 shows a summary of execution modes per source.Table 2Database search methodology.Table 2 dummy alt textDatabase APIWeb ApplicationLocal FilteringWoS Core✘✓✘Scopus✓✘✘ACM Digital Library✘✓✘IEEE Xplore✘✓✘Springer Link✓✘title, abstract, language, subjectScienceDirect✓✘✘ Table 2Database search methodology.Table 2 dummy alt textDatabase APIWeb ApplicationLocal FilteringWoS Core✘✓✘Scopus✓✘✘ACM Digital Library✘✓✘IEEE Xplore✘✓✘Springer Link✓✘title, abstract, language, subjectScienceDirect✓✘✘ Database search methodology. Deduplication. The search results (primary studies) retrieved from multiple sources must be deduplicated before proceeding with selection. Deduplication may be performed manually or automatically. This protocol strongly suggests performing the deduplication automatically whenever possible. The tool used during the pilot SMS provided automated deduplication using Dedupe3 as its deduplication engine, an open-source Python library. A tool like Dedupe is recommended because the use of standard identifiers (e.g. DOI) is not uniform across queried citation databases. In our SMS, an exact version of Dedupe was pinned as a dependency of MapwiseFox [12] to ensure reproducibility. As part of the protocol, the exact settings for running dedupe with the same parameters used when constructing the SMS must be preserved. Dedupe uses active learning to create the optimal configuration for a given dataset. When such a tool is used, the settings obtained through active learning must be provided as part of the reproduction package. Snowballing. Snowballing may be applied at various stages, depending on the desired outcome [3]. This protocol mandates applying it after the initial automated search and the first round of selection have both been completed and agreed upon. Both backward snowballing (tracing references cited by the included studies) and forward snowballing (identifying subsequent studies that cite them) must be performed. We consider a single level of snowballing sufficient but admit that the number of levels of snowballing must be determined on a case-by-case basis [2,3]. In the pilot SMS, one level of snowballing was executed automatically via MapwiseFox [12], which uses the Semantic Scholar API4 to retrieve citing and cited works. Semantic Scholar was selected for three reasons. First, it provides both the reference list and the citation list of a paper through a single API endpoint, which is a practical advantage over sources that require separate queries for each direction. Second, its coverage of computer science and data management literature is comprehensive; its recall is competitive with, and in some cases exceeds, that of the citation databases used in the primary search [13]. Third, its API is publicly accessible without institutional approval, which is consistent with the open and reproducible character of this protocol. Preprints and other non-peer reviewed literature are not a concern during snowballing because candidate studies are retrieved by their relationship to an already-included, peer-reviewed paper. Selection and inter-rater agreement for search results and snowballed candidates follow the procedure described in the dedicated section below (including inter-rater calibration and agreement). Studies already included in the results of the initial automated search are identified by their DOI and skipped automatically. Selection. The selection process identifies primary studies for data extraction from the candidate sets produced by automated search and snowballing. It proceeds in three phases. The first phase is applied to the deduplicated results of the initial automated search. The second phase applies the same criteria to candidates identified through snowballing that were not already evaluated in the first phase. The third and final phase uses quality assessment as a gate: studies that do not meet the minimum quality threshold are removed from the included set before data extraction begins. The inclusion and exclusion criteria are identical across all three phases. Inclusion Criteria. A candidate study is included if and only if it satisfies the following conditions:•Published within a clearly determined range.•Written in English.•Where multiple studies describe the same system, the study designated as primary by the authors is preferred. Where no such designation exists, the study that is most comprehensive with respect to software architecture concepts is selected, and the remaining studies describing the same system are excluded. •Published within a clearly determined range. Published within a clearly determined range. •Written in English. Written in English. •Where multiple studies describe the same system, the study designated as primary by the authors is preferred. Where no such designation exists, the study that is most comprehensive with respect to software architecture concepts is selected, and the remaining studies describing the same system are excluded. Where multiple studies describe the same system, the study designated as primary by the authors is preferred. Where no such designation exists, the study that is most comprehensive with respect to software architecture concepts is selected, and the remaining studies describing the same system are excluded. Exclusion Criteria. A candidate study is excluded if any of the following conditions hold:1.The title or abstract does not describe a software system, tool, application, framework, or architecture.2.The title or abstract does not describe a fully functional system.3.The topic of interest is presented as an instrument or an application within another field of study.4.The work focuses on a specific algorithm, technique, or experimental evaluation without describing the software in which it is embedded.5.The work is a secondary study: a systematic review, systematic mapping study, survey, or literature overview.6.The work is not peer-reviewed, including preprints, technical reports, and extended abstracts.7.The work does not attain the minimum quality threshold as determined through quality assessment. 1.The title or abstract does not describe a software system, tool, application, framework, or architecture. The title or abstract does not describe a software system, tool, application, framework, or architecture. 2.The title or abstract does not describe a fully functional system. The title or abstract does not describe a fully functional system. 3.The topic of interest is presented as an instrument or an application within another field of study. The topic of interest is presented as an instrument or an application within another field of study. 4.The work focuses on a specific algorithm, technique, or experimental evaluation without describing the software in which it is embedded. The work focuses on a specific algorithm, technique, or experimental evaluation without describing the software in which it is embedded. 5.The work is a secondary study: a systematic review, systematic mapping study, survey, or literature overview. The work is a secondary study: a systematic review, systematic mapping study, survey, or literature overview. 6.The work is not peer-reviewed, including preprints, technical reports, and extended abstracts. The work is not peer-reviewed, including preprints, technical reports, and extended abstracts. 7.The work does not attain the minimum quality threshold as determined through quality assessment. The work does not attain the minimum quality threshold as determined through quality assessment. Criteria 2 and 3 warrant exemplification. In the case of an entity resolution system, a primary study that does not describe an end-to-end entity resolution system should be excluded. More specifically, studies that address a single ER stage – a blocking method, a matching function, or a clustering algorithm – in isolation must be excluded. Additionally, primary studies where the mapped topic is not a primary concern must also be excluded. For ER systems, this criterion eliminates those primary studies that present them as instruments or applications within other fields of study. As an example, consider record linkage being presented as an epidemiological data-collection method or named entity linking contained as a step within another, more comprehensive natural language processing application. In these cases, the primary study cannot be focused on the software design of the ER system unless it contains multiple main themes. Such studies are excluded to avoid partial inclusions in the final map. Before quality assessment, the inclusion and exclusion criteria are applied primarily on the basis of the title and abstract. Where these are insufficient to reach a determination and during quality assessment, the full text must be consulted. Selection Procedure. We propose a selection procedure where one reviewer tackles the entire selection and other reviewers handle equal parts (for N reviewers, N-1 splits of the initial workload). From here on, we describe the selection procedure with three reviewers, exemplified in Fig. 1. Let us refer to the reviewers as R1, R2, and R3. Once an initial set of deduplicated search results is available, it is split among the three reviewers so that R1 receives the entire set, while R2 and R3 receive non-overlapping halves through random partitioning. R1 applies the inclusion and exclusion criteria independently to the full candidate set. R2 and R3 apply the same criteria independently to their allocated subset.Fig. 1Selection process with three reviewers.Fig 1 dummy alt text Selection process with three reviewers. The decisions of R2 and R3 are compared with R1 using the kappa score [14]. Following the original classification, a kappa value greater than 0.60 is taken to indicate substantial agreement. Therefore, if kappa exceeds 0.60, an agreement is established between the pair of reviewers. If kappa does not exceed 0.60, the reviewer pair must meet to discuss each point of disagreement and calibrate their understanding of the criteria or their application. The revised decisions replace the originals. These steps are repeated until substantial agreement is reached between R1 and the other team members. This protocol advises leniency towards inclusion when a paper's relevance is uncertain. This is a methodological choice to maximize recall and reduce the risk of excluding relevant studies, not a correction for the statistical properties of kappa. The team must conduct sessions for reviewing the resulting selection until consensus is reached. The consensus set is used as the input for snowballing. Candidates identified through snowballing that were not previously evaluated undergo the same procedure. The final included set, after both search-based and snowballing-based selection, is then submitted to quality assessment. Quality Assessment. Table 3 presents a scoring card for assessing the primary studies that participate in the systematic mapping study. The scoring card is heavily inspired from [15], but also draws on other checklists for reading and reviewing case studies [16]. The checklist examples found there are tailored to cater to studies centered around software architecture. The resulting checklist uses four axes for measuring the quality of a study: reporting, rigor, credibility, and relevance [15].Table 3Primary study quality assessment criteria.Table 3 dummy alt textCategoryNumberQuestionDescriptionScoringReportingre1Is the study methodical and formal? Does it explicitly follow literature guidelines; does it explicitly mention a methodology? Tone.Academic formalism and neutral tone yield higher scores according to this criterion.1–10re2Are the research objectives clear?Clearly stated research questions or hypotheses and well-presented motives are the north star for this criterion.1–10re3Is the description of the research method adequate?The research method should describe itself clearly. Steps taken to achieve goals must be unambiguous. A positive is when methodology is explained and subsequently applied according to that explanation.1–10Rigorri1Is the research design appropriate?The score should reflect how fit the research methodology is in relation to the research objectives. The presence of discussions around "bias" and "limitations" are positive.1–10ri2Were the amount and diversity of data sources used in the experimental part of the paper sufficient in relation to the research objectives?Positives: triangulation, data relevance, and adequacy in relation to research objectives and supporting conclusions.1–10Credibilityc1Is there a clear statement of findings linked to evidence?Possible positives: presence of discussion sections, triangulation, and evidence-supported conclusions. The presence of a key findings section should be considered a good omen.1–10Relevancer1How relevant is the study for research in software architecture?Evaluate based on presence, depth, and importance of architecture concepts. Process-related diagram at a bare minimum. Positives: section dedicated to software architecture, static component design, architecture principles, discussion around architecture trade-offs (e.g., performance vs extensibility or scaling the solution); references to software architecture works in the related work, background or discussion sections of the paper; mentions of practical applications/implications of the research in practice that have an impact/generate software architecture.1–10r2How generalizable are the conclusions of the study?What is the likelihood you would be able to build a software system that addresses the mapped topic for your own needs based on the design presented in the paper? E.g., is the primary study conducive towards building a system beyond the specifics described in the study?1–10 Table 3Primary study quality assessment criteria.Table 3 dummy alt textCategoryNumberQuestionDescriptionScoringReportingre1Is the study methodical and formal? Does it explicitly follow literature guidelines; does it explicitly mention a methodology? Tone.Academic formalism and neutral tone yield higher scores according to this criterion.1–10re2Are the research objectives clear?Clearly stated research questions or hypotheses and well-presented motives are the north star for this criterion.1–10re3Is the description of the research method adequate?The research method should describe itself clearly. Steps taken to achieve goals must be unambiguous. A positive is when methodology is explained and subsequently applied according to that explanation.1–10Rigorri1Is the research design appropriate?The score should reflect how fit the research methodology is in relation to the research objectives. The presence of discussions around "bias" and "limitations" are positive.1–10ri2Were the amount and diversity of data sources used in the experimental part of the paper sufficient in relation to the research objectives?Positives: triangulation, data relevance, and adequacy in relation to research objectives and supporting conclusions.1–10Credibilityc1Is there a clear statement of findings linked to evidence?Possible positives: presence of discussion sections, triangulation, and evidence-supported conclusions. The presence of a key findings section should be considered a good omen.1–10Relevancer1How relevant is the study for research in software architecture?Evaluate based on presence, depth, and importance of architecture concepts. Process-related diagram at a bare minimum. Positives: section dedicated to software architecture, static component design, architecture principles, discussion around architecture trade-offs (e.g., performance vs extensibility or scaling the solution); references to software architecture works in the related work, background or discussion sections of the paper; mentions of practical applications/implications of the research in practice that have an impact/generate software architecture.1–10r2How generalizable are the conclusions of the study?What is the likelihood you would be able to build a software system that addresses the mapped topic for your own needs based on the design presented in the paper? E.g., is the primary study conducive towards building a system beyond the specifics described in the study?1–10 Primary study quality assessment criteria. In the original SMS methodology, quality assessment is optional and discouraged because the objective is to broaden the mapping spectrum [1]. This SMS protocol advocates for the inclusion of quality assessment for mapping studies targeting software architecture because during preliminary scoping for [11] it was discovered that finding ER system descriptions would prove difficult and too subjective if only established techniques in study mapping (e.g. keywording, automated search, reading abstracts or skimming the full article text) were used. Therefore, the SMS protocol was updated to include quality assessment as a mandatory exclusion gate. The presence of a quality assessment exclusion gate introduces significant selection bias, but it also significantly increases the relevance of the primary studies comprising the final map. As an example, when searching for entity resolution system descriptions, these are not uniformly produced. A paper may describe an end-to-end ER process, pass every keyword-based inclusion criterion, and still provide insufficient information to reason about how the system is structured, deployed, or extended. Another situation that was encountered in practice concerns selection based on title and abstract alone. The result contained numerous position papers and short papers that did not describe software architectures of functioning systems, that were returned via automated search even with exclusion filters based on content type in place. Because the title and abstract were highly relevant, these papers were retained during the initial two selection rounds (after search and after snowballing). The conclusion was that determining whether a study provides actionable architectural knowledge cannot be resolved by reading its title and abstract alone. At the very least, the rigor of its methodology, the credibility of its findings, the adequacy of its evidence base, and its relevance to software architecture as a design concern must be assessed. Table 3 captures the SMS quality assessment criteria. The rationale for using quality assessment as an exclusion gate rests on the nature of the map being constructed. For a practitioner attempting to build a comparable system, a primary study that lacks credible findings, evaluates its system on a single opaque dataset, or omits any discussion of architectural decisions might raise more questions than it answers. Excluding such studies on quality grounds is not a narrowing of the evidence base in the sense intended by [1]; it is a recognition that the unit of value in an SMS conducted using this protocol is not the existence of a study but the actionable architectural knowledge it contains. A corpus of fewer, higher-quality studies provides a more reliable map than a larger corpus padded with studies that report insufficient detail to support design inference. Quality Assessment Process. Fig. 2 illustrates the quality assessment process for primary studies. The final selection is partitioned into N + 1 parts across N human reviewers and one LLM-based reviewer. Each human reviewer is assigned one of the N parts, while the remaining part is assessed using large language models (LLMs).Fig. 2Primary study quality assessment process.Fig 2 dummy alt text Primary study quality assessment process. The number of reviewers N should be chosen with respect to both the total number of studies and the quality assessment instrument being used. This protocol advocates exactly three assessments per study, which combined with a simple round-robin assignment across N + 1 reviewers (N human, one LLM) uniquely determines the workload per reviewer. For 120 studies and N = 4, this yields 72 studies per reviewer, with the 72 primary studies contained in the LLM partition always receiving exactly two additional human assessments. In our pilot SMS, this constraint was operationalized using the split-workload for-evaluation command of MapwiseFox [12]. Round-robin is suggested because it does not fix which human reviewers overlap with the LLMs. Therefore, the overlap can be treated as a random effect and should be quantified using methods that take random effects into account rather than metrics that assume fixed reviewer pairings. More generally, we recommend partitioning schemes where each study must be assessed by three reviewers, and the overlap between human and LLM reviewers is not predetermined. Three raters per study is recommended as it balances bias reduction and scoring robustness: a single dissenting score can be contextualized against two others without the overhead and diminishing returns of four or more raters. Researchers adopting alternative partitioning strategies should verify that the constraints are preserved, and should report their chosen scheme explicitly to ensure reproducibility. The ratio of human to LLM reviewers is likewise flexible, provided the random effect is maintained. Using multiple LLMs or an entirely LLM-based panel is possible; however researchers should note that the composition of the reviewer panel affects the interpretation of ICC and related metrics: a mixed human-LLM panel speaks to human-machine agreement, whereas a homogeneous panel speaks only to consistency within that reviewer type. N parts are assessed by human reviewers, while part N + 1 of the selection is assessed by using large language models (LLMs). This workload distribution ensures that each primary study is assessed independently at least twice by humans. The process starts with human evaluators grading primary studies. Four basic statistics for the quality ratings provided by the human evaluators are computed per evaluation criterion: the minimum, maximum, and average scores. The next section provides the details for selecting LLM evaluators for each of the criteria listed in Table 3 based on these statistics. Evaluation Using LLMs. A portion of the primary studies are assessed by large language models to improve scalability and provide assessments that are statistically independent of human reviewer subjectivity. No specific model selection is prescribed, but the following principles should guide it:•Open-weights representation: at least one open-weights model must be included. This ensures the assessment procedure is partially reproducible without API access or cost.•Vendor diversity: models from multiple vendors should be selected to reduce the risk that any single vendor's training process or scoring behavior dominates the results.•Parameter scale diversity: including models of different sizes allows post-hoc analysis of whether model scale affects scoring reliability and informs cost-conscious replication.•Intra-vendor comparison: where a vendor offers both a standard and a lightweight variant, including both allows assessment of whether the lighter variant produces comparable scores. This informs future practitioners on the minimum viable model tier.•Context window adequacy: every selected model must support a context window large enough to ingest the longest primary studies in the candidate set without truncation.•Deployment platform diversity: including models accessed via local inference, managed cloud, and direct vendor APIs tests whether deployment context affects reproducibility.•Default inference parameters: all models should be queried using their default inference parameters. No settings, such as temperature, should be modified. If custom inference parameters are used, these must be documented and archived together with the implementation to allow independent verification.•Assessment date recording: the date on which each model was queried should be recorded. Model behavior can change across deployment updates even when the version string is fixed. •Open-weights representation: at least one open-weights model must be included. This ensures the assessment procedure is partially reproducible without API access or cost. Open-weights representation: at least one open-weights model must be included. This ensures the assessment procedure is partially reproducible without API access or cost. •Vendor diversity: models from multiple vendors should be selected to reduce the risk that any single vendor's training process or scoring behavior dominates the results. Vendor diversity: models from multiple vendors should be selected to reduce the risk that any single vendor's training process or scoring behavior dominates the results. •Parameter scale diversity: including models of different sizes allows post-hoc analysis of whether model scale affects scoring reliability and informs cost-conscious replication. Parameter scale diversity: including models of different sizes allows post-hoc analysis of whether model scale affects scoring reliability and informs cost-conscious replication. •Intra-vendor comparison: where a vendor offers both a standard and a lightweight variant, including both allows assessment of whether the lighter variant produces comparable scores. This informs future practitioners on the minimum viable model tier. Intra-vendor comparison: where a vendor offers both a standard and a lightweight variant, including both allows assessment of whether the lighter variant produces comparable scores. This informs future practitioners on the minimum viable model tier. •Context window adequacy: every selected model must support a context window large enough to ingest the longest primary studies in the candidate set without truncation. Context window adequacy: every selected model must support a context window large enough to ingest the longest primary studies in the candidate set without truncation. •Deployment platform diversity: including models accessed via local inference, managed cloud, and direct vendor APIs tests whether deployment context affects reproducibility. Deployment platform diversity: including models accessed via local inference, managed cloud, and direct vendor APIs tests whether deployment context affects reproducibility. •Default inference parameters: all models should be queried using their default inference parameters. No settings, such as temperature, should be modified. If custom inference parameters are used, these must be documented and archived together with the implementation to allow independent verification. Default inference parameters: all models should be queried using their default inference parameters. No settings, such as temperature, should be modified. If custom inference parameters are used, these must be documented and archived together with the implementation to allow independent verification. •Assessment date recording: the date on which each model was queried should be recorded. Model behavior can change across deployment updates even when the version string is fixed. Assessment date recording: the date on which each model was queried should be recorded. Model behavior can change across deployment updates even when the version string is fixed. Table 4 inventories an example selection of LLMs following these principles, without specifying the assessment date.Table 4Example selection of LLMs to be used in the quality assessment process. Open-weights and closed weights models, from different vendors, of various parameter scales and using more than one deployment platform should be included. The model version should be specified.Table 4 dummy alt textNameVendorOpen# ParamsContext Window SizePlatformqwen3:8bAlibaba✓8×10932,768Local (ollama1)gpt-5-mini-2025-08-07OpenAI✘undisclosed400,000OpenAI API2claude-haiku–4-5-20251001Anthropic✘undisclosed200,000AWS Bedrock3gemini-2.5-flashGoogle✘undisclosed1MGoogle AI Studio41https://ollama.com/.2https://openai.com/api/.3https://aws.amazon.com/bedrock/.4https://aistudio.google.com/. Table 4Example selection of LLMs to be used in the quality assessment process. Open-weights and closed weights models, from different vendors, of various parameter scales and using more than one deployment platform should be included. The model version should be specified.Table 4 dummy alt textNameVendorOpen# ParamsContext Window SizePlatformqwen3:8bAlibaba✓8×10932,768Local (ollama1)gpt-5-mini-2025-08-07OpenAI✘undisclosed400,000OpenAI API2claude-haiku–4-5-20251001Anthropic✘undisclosed200,000AWS Bedrock3gemini-2.5-flashGoogle✘undisclosed1MGoogle AI Studio41https://ollama.com/.2https://openai.com/api/.3https://aws.amazon.com/bedrock/.4https://aistudio.google.com/. Example selection of LLMs to be used in the quality assessment process. Open-weights and closed weights models, from different vendors, of various parameter scales and using more than one deployment platform should be included. The model version should be specified. https://ollama.com/. https://openai.com/api/. https://aws.amazon.com/bedrock/. https://aistudio.google.com/. The LLM prompts used for evaluating the primary studies must contain instructions that are identical to those received by the human reviewers. In the pilot SMS, we used MapwiseFox [12] with zero-shot prompting based on standardized templates [17] to enforce this requirement. The templates used in our SMS are available in the MapwiseFox code [12]. An example configuration covering the quality assessment criteria listed in Table 3 is provided as supplementary material in Listing 1.Listing 1Example MapwiseFox JSON configuration for running automated quality assessment using LLMs.Listing 1 dummy alt text{"topic": "software architecture of entity resolution systems","criteria": [{"label": "re1","category": "reporting","question": "Is the study methodical and formal?","description": "Evaluate tone and language.\nFormal studies follow widely accepted formatting guidelines.\nExplicit mention of the used methodology is good.\nA introduction-related work-conclusions structure scores higher than improvised structure.\nAcademic tone scores better than informal tone.","scoring": "1 to 10, higher is more formal and better structured"},{"label": "re2","category": "reporting","question": "Are the research objectives clear?","description": "Look for clearly structured research questions or hypotheses and well-articulated research objectives.","scoring": "1 to 10, higher is more readable"},{"label": "re3","category": "reporting","question": "Is the description of the research method adequate?","description": "Clear description of steps taken. Methodology is explained and applied according to the explanation.","scoring": "1 to 10, higher means the research method is more adequate"},{"label": "ri1","category": "rigour","question": "Is the research design appropriate?","description": "The baseline is that the methodology is appropriate for attaining the research objectives.\nPresenting biases and limitations is better.\nA \"Discussion\" section improves the score.","scoring": "1 to 10, higher indicates a more appropriate research design"},{"label": "ri2","category": "rigour","question": "Was the amount and diversity of data sources used in the experimental part of the paper sufficient in relation to the research objectives?","description": "Number and diversity of data sources drives the score.\nMultiple business domains or multiple data modalities increase the score.\nTriangulation increases the score.\nDebating data relevance and adequacy in relation to research objectives is good.","scoring": "1 to 10, higher values indicate a better data coverage for the research objectives"},{"label": "c1","category": "credibility","question": "Was there a clear statement of findings linked to evidence?","description": "The presence of a discussion section is good. Triangulation and evidence-supported conclusions are also desirable. The explicit presence of a \"Key Findings\" section is good.","scoring": "1 to 10, higher values mean the paper is more credible"},{"label": "r1","category": "relevance","question": "How relevant is the study for research in software architecture?","description": "Description of a system is a starting point.\nThe description of an architecture and architectural concepts is good.\nProviding multiple perspectives (process, layout, use case, deployment and so on) is good.\nA dedicated section for software or system architecture is good.\nUsing an architecture methodology (UML, Archimate, TOGAF, ATAM, C4, 4 + 1 and so on) is good.","scoring": "1 to 10, higher values mean that software architecture is covered better"},{"label": "r2","category": "relevance","question": "How generalizable are the conclusions of the study?","description": "Look for how-tos or similar instructions that practitioners to build an ER system based on the study. Bad omen: authors explicitly limit the scope of the applications to the business/research domain (e.g. only used for newspaper data, social media, in a certain language, etc.). Do not imagine use cases outside what the paper explicitly describes!","scoring": "1 to 10, higher values indicate that it is easy to build an ER system based on the information contained in the paper"}]} Listing 1Example MapwiseFox JSON configuration for running automated quality assessment using LLMs.Listing 1 dummy alt text{"topic": "software architecture of entity resolution systems","criteria": [{"label": "re1","category": "reporting","question": "Is the study methodical and formal?","description": "Evaluate tone and language.\nFormal studies follow widely accepted formatting guidelines.\nExplicit mention of the used methodology is good.\nA introduction-related work-conclusions structure scores higher than improvised structure.\nAcademic tone scores better than informal tone.","scoring": "1 to 10, higher is more formal and better structured"},{"label": "re2","category": "reporting","question": "Are the research objectives clear?","description": "Look for clearly structured research questions or hypotheses and well-articulated research objectives.","scoring": "1 to 10, higher is more readable"},{"label": "re3","category": "reporting","question": "Is the description of the research method adequate?","description": "Clear description of steps taken. Methodology is explained and applied according to the explanation.","scoring": "1 to 10, higher means the research method is more adequate"},{"label": "ri1","category": "rigour","question": "Is the research design appropriate?","description": "The baseline is that the methodology is appropriate for attaining the research objectives.\nPresenting biases and limitations is better.\nA \"Discussion\" section improves the score.","scoring": "1 to 10, higher indicates a more appropriate research design"},{"label": "ri2","category": "rigour","question": "Was the amount and diversity of data sources used in the experimental part of the paper sufficient in relation to the research objectives?","description": "Number and diversity of data sources drives the score.\nMultiple business domains or multiple data modalities increase the score.\nTriangulation increases the score.\nDebating data relevance and adequacy in relation to research objectives is good.","scoring": "1 to 10, higher values indicate a better data coverage for the research objectives"},{"label": "c1","category": "credibility","question": "Was there a clear statement of findings linked to evidence?","description": "The presence of a discussion section is good. Triangulation and evidence-supported conclusions are also desirable. The explicit presence of a \"Key Findings\" section is good.","scoring": "1 to 10, higher values mean the paper is more credible"},{"label": "r1","category": "relevance","question": "How relevant is the study for research in software architecture?","description": "Description of a system is a starting point.\nThe description of an architecture and architectural concepts is good.\nProviding multiple perspectives (process, layout, use case, deployment and so on) is good.\nA dedicated section for software or system architecture is good.\nUsing an architecture methodology (UML, Archimate, TOGAF, ATAM, C4, 4 + 1 and so on) is good.","scoring": "1 to 10, higher values mean that software architecture is covered better"},{"label": "r2","category": "relevance","question": "How generalizable are the conclusions of the study?","description": "Look for how-tos or similar instructions that practitioners to build an ER system based on the study. Bad omen: authors explicitly limit the scope of the applications to the business/research domain (e.g. only used for newspaper data, social media, in a certain language, etc.). Do not imagine use cases outside what the paper explicitly describes!","scoring": "1 to 10, higher values indicate that it is easy to build an ER system based on the information contained in the paper"}]} Example MapwiseFox JSON configuration for running automated quality assessment using LLMs. After performing the evaluations, each LLM assessment is compared with the human assessments for the same articles using four metrics: mean absolute error (MAE), root mean squared error (RMSE), intraclass correlation coefficient (ICC) [18], and Lin’s concordance correlation coefficient (CCC) [19]. MAE, RMSE, and CCC are pairwise metrics computed between the LLM score vector y of length n and a reference vector x of the same length. The reference x is one of three statistics computed on the human ratings: the average, the minimum, or the maximum. MAE measures the mean absolute difference between y and x:MAE(x,y)=1n∑i=1n|yi−xi| RMSE penalizes large individual errors more than MAE does through its quadratic term:RMSE(x,y)=1n∑i=1n(yi−xi)2 Both metrics return values in [0, 10] when scores are drawn from the same interval. A model whose RMSE exceeds its MAE by a wide margin should be interpreted as producing more extreme outlier scores. Lin’s CCC measures agreement by combining correlation (precision) with bias (accuracy). It penalizes both scale/shift differences and the lack of linear association. Given the means x¯, y¯, the variances sx2, sy2, and the covariance between x and y, sxy of the paired vectors, the CCC is:CCC=2sxysx2+sy2+(x¯−y¯)2 The CCC takes values in [−1, 1]. The interpretation of the CCC varies in literature. Values above 0.8 are viewed as acceptable, and a lower bound of 0.2 is recommended in earlier texts; later work classifies all values below 0.9 as poor. This protocol does not advise interpreting CCC in isolation; instead, it should be paired with MAE, RMSE, and ICC. The ICC [18] captures variation across all three ratings that are available for each primary study simultaneously, including the LLM rating. LLMs were observed to produce different scores between runs and must be treated as a random effect. On the other hand, because the workload partitioning scheme does not allow any single human rater to assess the same papers as the LLM, the human component is treated as a random effect, too. This motivates using the two-way random effects, absolute agreement model [20]:ICC(2,1)=MSR−MSEMSR+(k−1)·MSE+kn·(MSC−MSE),where MSR is the mean square between subjects, MSC the mean square between raters, MSE the mean square error, n the number of papers, and k the number of raters. The ICC takes values in [0, 1], with values being considered poor below 0.5, moderate in the interval [0.5, 0.75), good in the interval [0.75, 0.9) and excellent between 0.9 and 1 [20]. To support an overall ranking we suggest a ranking device combining the four metrics into a single score.OverallScore=CCC+ICC(2,1)MAE×RMSE This formula uses the CCC, MAE and RMSE computed against the average human ratings and is a ranking heuristic, not a validated measurement instrument. The formula balances CCC and ICC equally, which is appropriate when no prior reason favors concordance over reliability. Assigning more weight to one metric or the other is defensible in settings where that metric is known to matter more. For example, CCC-heavy weighting is favorable when detecting systematic rater bias is the priority. ICC-heavy weighting is a good choice when the consistency of rank ordering across raters takes precedence. Our own sensitivity analysis (see Method Validation, §Composite-Score Sensitivity (LLM Selection)) shows that CCC-heavy and ICC-heavy weightings produce near-identical final selections (Jaccard 1.0 between the two weighted variants), so the equal-weighting choice has limited practical consequence for the protocol's outcome. The formula is undefined when both error metrics approach zero, which would require an LLM to reproduce the average of the human ratings exactly. Given the granularity of the scoring scale, the assumption that inter-rater variability among human reviewers exists, and the inherent subjectivity of quality assessment, this case is extremely hard to realize in practice. As a mitigation for this case we propose instituting a saturation threshold of 1.0. A composite score exceeds 1.0 if and only if the numerator is greater than the denominator. Since the numerator is bounded above by 2, this requires MAE×RMSE<2. In turn, this implies that both MAE<2≈1.41 and RMSE<2. This would place the errors around 14% of the full scoring range on a ten point scale. Any model achieving average errors below 2 would already be performing well enough for the purpose of using it as a quality assessment assistant. Therefore, the composite score of 1.0 can serve as a natural saturation threshold beyond which differences in rank ordering carry no practical significance. The structure for the overall score reflects three deliberate choices. The numerator sums concordance and reliability; models that score well on both are rewarded more than models that excel on only one. A different weighting of concordance and reliability can be employed (e.g. CCC+ICC(2,1)MAE×RMSE) while still maintaining this quality. Second, the denominator uses the product of MAE and RMSE rather than their sum. Because RMSE ≥ MAE always holds, and because RMSE penalizes outliers more than MAE does, their ratio RMSE/MAE measures the outlier character of the error distribution. The product MAE×RMSE can therefore be read as a MAE-scaled penalty that grows with both average error and error variability, while remaining less extreme than RMSE2 at the upper end of the observed range. Finally, no formal derivation is claimed for this formula; it is offered as a device for collapsing four metrics into a single rank ordering, and we instead report a sensitivity analysis of its behaviour (see Method Validation, §Composite-Score Sensitivity (LLM Selection)). This ranking device can be replaced with another choice, in which case that choice must be documented in the SMS introducing it. For each quality assessment criterion, the scores of the highest-ranked model are retained. This motivates maintaining as wide a selection of LLMs as possible, as a larger pool increases the likelihood of retaining the strongest assessment for each criterion. Finalizing Quality Assessment. With three raters per study, the protocol requires a rule for collapsing their scores into a single aggregate per criterion. For a different number of raters, the SMS authors must select and document an aggregation statistic appropriate to that count. The protocol does not prescribe a single aggregation statistic such as the simple mean, a weighted average or the median. The only requirement is to document the chosen aggregation in the SMS that uses it. The protocol also allows gating the selection at the category level (i.e. the category-level averages across criterion-level aggregated scores must meet the threshold) or at the criterion level (i.e. the aggregated criterion-level scores themselves must meet it). Where quality assessment criteria are grouped into categories (as this protocol prescribes), category-level gating should be the default because it is more lenient and preserves studies where one criterion is weak but the category as a whole is adequate. This guidance is consistent with the SMS methodology's objective of broadening the mapping spectrum [1]. We recommend a weighted average inspired by the Program Evaluation and Review Technique (PERT) [21]. Originally devised for project scheduling, PERT combines three-point estimates – optimistic (O), pessimistic (P) and most likely (M) – into a weighted average: PERT=(O+4×M+P)6. With exactly three raters per study, their scores map naturally to these three roles: the median score becomes M, and the remaining scores serve as O and P. Because the formula treats O and P symmetrically, the assignment of the two non-median scores to these roles is inconsequential and the mapping is fully determined by rank order alone. Compared to simpler alternatives, PERT occupies a deliberate middle ground. The simple mean weights all raters equally, making it sensitive to outliers – a concern when one rater is an LLM whose scores may be systematically higher or lower. The median (equivalently, the trimmed mean for three raters) discards the two extreme scores entirely, potentially losing information when raters disagree. PERT retains all three scores while assigning four times the weight to the central value, balancing robustness with inclusiveness. In our mapping study the mean and median produced identical selections, and PERT differed from them by nine studies (Jaccard 0.848). This property is what motivates recommending PERT over both alternatives, while acknowledging that the mean and median remain defensible choices. We suggest excluding primary studies with aggregate scores below 5, a threshold derived from our pilot score distribution [11]. This threshold should not be treated as absolute; researchers are encouraged to relax or tighten it as warranted by the distribution of scores in their own corpus. In general, appropriate thresholds can be determined by plotting the score distribution across all screened studies and identifying natural break points, a technique consistent with standard practice in data-driven quality assessment [3]. The value of 5 was chosen for our pilot [11] because it sits in a stable region of the score distribution, and is offered here as a starting point, not a prescriptive rule. Researchers should verify that their chosen threshold sits in a stable region of their own score distribution before adopting it. Data Extraction. Table 5 presents a list of features which should be extracted from each primary study. This list should be regarded as an example used for mapping out the architecture of entity resolution systems. It can be adapted to mapping studies on software architecture in other domains. Two independent reviewers must read each paper and record the feature values. Extracted values are then merged according to the merge criteria listed for each feature. Unique means that these features have only one valid value in the corpus of primary studies. Union means all extracted values are kept. Discussion means the two reviewers must agree on a single value. A third reviewer resolves any disagreement that the pair cannot settle. Dissenting opinions are recorded.Table 5Features extracted from each primary study.Table 5 dummy alt textNoNameDescriptionExtr.Merge1TitleStudy titleManualUnique2AuthorsAuthors of the studyManualUnique3PublicationName of journal or conferenceManualUnique4Publication TypeJournal, conference, or other venueManualUnique5Publication DateYear of publicationManualUnique6Evidence TypeSystem, tool, application, framework, or architectureManualUnion7Arch. RepresentationNone, informal (boxes and lines, C4, etc.), or formal (UML, ArchiMate, etc.). Default: noneManualDiscussion8Design ElementsDesign principles, patterns, implementation standards, or deployment strategies. Default: noneManualUnion9Architecture StyleStyles per Chapter 9 of Richards and Ford (2020), or none. Default: none.ManualDiscussion10Primary TechnologyMain programming language, or none for abstract architecture. Default: noneManualUnion Table 5Features extracted from each primary study.Table 5 dummy alt textNoNameDescriptionExtr.Merge1TitleStudy titleManualUnique2AuthorsAuthors of the studyManualUnique3PublicationName of journal or conferenceManualUnique4Publication TypeJournal, conference, or other venueManualUnique5Publication DateYear of publicationManualUnique6Evidence TypeSystem, tool, application, framework, or architectureManualUnion7Arch. RepresentationNone, informal (boxes and lines, C4, etc.), or formal (UML, ArchiMate, etc.). Default: noneManualDiscussion8Design ElementsDesign principles, patterns, implementation standards, or deployment strategies. Default: noneManualUnion9Architecture StyleStyles per Chapter 9 of Richards and Ford (2020), or none. Default: none.ManualDiscussion10Primary TechnologyMain programming language, or none for abstract architecture. Default: noneManualUnion Features extracted from each primary study. Reviewers should be selected based on their experience on the SMS topic. In the case of the related SMS, software architecture and software engineering research was required. The team should include people with varying degrees of experience, from PhD students to senior researchers. When the topic of the mapping study requires it, initial calibration sessions should be performed. Each feature in Table 5 lists a default value. This default was chosen to reflect common values observed in practice during preliminary scoping for the SMS performed based on this methodology. Default values should be used only when a feature is not addressed in the primary study. The use of default values does not imply that the primary study is deficient. Table 5 also lists features whose definitions require additional explanation. The evidence type feature describes the form of the software contribution reported in a primary study. The following five values are used as working definitions for this study. A system is a complete, deployed or deployable software product that performs entity resolution end-to-end. A tool is a standalone executable or command-line utility that supports one or more ER tasks. A framework is a reusable software structure that provides extension points for building or configuring a software system. An application is a domain-specific implementation built on top of an existing platform or framework. An architecture is a structural description of an ER system – capturing its elements, relationships, and design principles (ISO 42010:2022) – with an optional accompanying implementation. A study may report more than one evidence type; in that case all applicable values are recorded. Architecture styles are general and reusable solutions to recurring structural problems in software systems. This study classifies architectures using the taxonomy introduced by [22] which covers layered, pipeline, microservices, event-driven, space-based, and microkernel styles, among others. Systems without explicit architectural organization are classified as none, a category that encompasses so-called big-ball-of-mud designs and simple unitary implementations. Data Extraction Process. The extraction procedure has three stages. First, each reviewer receives an allocated portion of the papers that passed quality assessment. The primary studies are partitioned in such a way that each portion receives a pair of evaluators. Second, reviewers independently extract all features for their assigned papers using standardized extraction forms. Third and finally, for each primary study, extracted data is merged item-wise according to the specified merge criteria. Discussion-based features require reviewers to resolve discrepancies through consensus, with a third reviewer adjudicating cases where agreement cannot be reached. Inter-rater agreement is measured using the Jaccard similarity coefficient. For a given paper and feature, the Jaccard coefficient between two evaluators is the size of the intersection of their extracted value sets divided by the size of their union. A coefficient of 1 indicates identical extractions; a coefficient of 0 indicates no overlap. The second and third steps are repeated until the extractions on the discussion-based features converge completely (Jaccard=1.0). Systematic Map. The output of the data extraction process constitutes the final dataset used to construct the systematic map. The systematic map is constructed using the baseline methodology [1,2]. Method validation The current method is only an extension of the established methodology for performing systematic mapping studies in software engineering [1,2]. Most of the extensions convert accepted optional steps (e.g., quality assessment) into mandatory ones for reasons that are already acknowledged as valid in the original methodology. The protocol suggests that systematic mapping studies devise a self-assessment checklist. While this is not new, the checklist structure is inspired by PRISMA (which is primarily used for SLRs). However, the contents of the checklist shown in Table 6 are inspired by the SMS methodology.Table 6The SMS self-assessment checklist should be filled in by SMS authors before publication.Table 6 dummy alt textSection#CriterionPageTitle & AbstractTitle1Report identifies the study as a systematic mapping study.Abstract2Structured abstract states: background, research questions, search strategy, inclusion criteria, number of primary studies, classification scheme, and key findings.IntroductionRationale3Motivation for mapping the research area is stated; relation to existing reviews is discussed.Objectives4Research questions are stated explicitly; they address scope, trends in article publication, and research methods used in the field.MethodsEligibility5Inclusion and exclusion criteria are specified (topic relevance, venue type, period, language).Sources6All databases and digital libraries searched are listed with the date of the last search.Search string7Full search strings are reported for each source; development process and validation test set are described.Selection8A two-stage screening process is described; number of reviewers and conflict-resolution procedures are stated.Keywording9Keywording processes and resulting classification schemes (dimensions and categories) are described.Extraction10Data items extracted from each primary study are listed; extraction procedures are described.Quality appraisal11Criteria used to assess relevance to software architecture are stated; appraisal procedure and outcome are reported.ResultsFlow diagram12Flow diagram shows record counts at each selection stage.Characteristics13Distribution of primary studies by year, venue, research type, and research method is reported.Classification14Results of the classification scheme are presented using appropriate visualizations (e.g., bubble plots, bar charts).Trends & gaps15Research trends and open gaps identified by the mapping are summarized.DiscussionInterpretation16aResults are interpreted in the context of research questions and existing knowledge.Limitations16bLimitations of the mapping process (e.g. regarding completeness or the subjectivity of classification) are discussed.Implications16cImplications for researchers and practitioners are stated.OtherProtocol17Existence of a mapping protocol and any deviations from it are reported.Funding18Funding sources and role of funders are declared.Conflicts19Competing interests of all authors are declared.Data20Availability of extraction forms, included/excluded study lists, and scripts is stated. Table 6The SMS self-assessment checklist should be filled in by SMS authors before publication.Table 6 dummy alt textSection#CriterionPageTitle & AbstractTitle1Report identifies the study as a systematic mapping study.Abstract2Structured abstract states: background, research questions, search strategy, inclusion criteria, number of primary studies, classification scheme, and key findings.IntroductionRationale3Motivation for mapping the research area is stated; relation to existing reviews is discussed.Objectives4Research questions are stated explicitly; they address scope, trends in article publication, and research methods used in the field.MethodsEligibility5Inclusion and exclusion criteria are specified (topic relevance, venue type, period, language).Sources6All databases and digital libraries searched are listed with the date of the last search.Search string7Full search strings are reported for each source; development process and validation test set are described.Selection8A two-stage screening process is described; number of reviewers and conflict-resolution procedures are stated.Keywording9Keywording processes and resulting classification schemes (dimensions and categories) are described.Extraction10Data items extracted from each primary study are listed; extraction procedures are described.Quality appraisal11Criteria used to assess relevance to software architecture are stated; appraisal procedure and outcome are reported.ResultsFlow diagram12Flow diagram shows record counts at each selection stage.Characteristics13Distribution of primary studies by year, venue, research type, and research method is reported.Classification14Results of the classification scheme are presented using appropriate visualizations (e.g., bubble plots, bar charts).Trends & gaps15Research trends and open gaps identified by the mapping are summarized.DiscussionInterpretation16aResults are interpreted in the context of research questions and existing knowledge.Limitations16bLimitations of the mapping process (e.g. regarding completeness or the subjectivity of classification) are discussed.Implications16cImplications for researchers and practitioners are stated.OtherProtocol17Existence of a mapping protocol and any deviations from it are reported.Funding18Funding sources and role of funders are declared.Conflicts19Competing interests of all authors are declared.Data20Availability of extraction forms, included/excluded study lists, and scripts is stated. The SMS self-assessment checklist should be filled in by SMS authors before publication. The initial design of this protocol was created by a single researcher. After the initial draft, the protocol was reviewed by senior researchers and adjusted iteratively in incremental steps. Lastly, the protocol was validated in practice with an SMS on the software architecture of ER systems [11]. The outlined method for conducting primary study evaluation using LLMs was the primary validation target. Composite-Score Sensitivity (LLM Selection). The composite score determines which LLM is selected per criterion, which in turn drives the final QA selection. Therefore, we report a sensitivity analysis of that choice. In our pilot SMS, we compared the equal-weights composite heuristic against its two weighted variants (CCC-heavy and ICC-heavy), each of the four single metrics (MAE, RMSE, CCC, ICC), and an equal-weight Borda count of the four metrics, feeding each rule's per-criterion winning model into the full protocol at threshold 5.0. The resulting selections were compared using the Jaccard coefficient, Spearman rho and individual studies added and removed from the baseline. Under mean aggregation, the equal-weights composite selected 58 studies; the two weighted variants and the CCC-only rule each removed one study (Jaccard 0.982, Spearman rho 0.953), and the Borda count reproduced the baseline selection exactly (Jaccard 1.0, Spearman rho 0.933). Under PERT aggregation, the equal-weights composite again selected 58 studies; the two weighted variants and the CCC-only rule each removed three studies (Jaccard 0.948, Spearman rho 0.952), while the Borda count was the closest non-composite alternative, removing two studies (Jaccard 0.966, Spearman rho 0.885). In general, the two weighted variants produced the same selection as each other across all five rater score aggregations (Jaccard 1.0) and changed the winning model on only two of the eight quality assessment criteria, with a maximum composite-score difference below 0.41. Single-metric rules were less stable: MAE-only added three studies under PERT (Jaccard 0.951, Spearman rho 0.833), and under the maximum-statistic robustness check removed fourteen studies (Jaccard 0.837). Across all five aggregation statistics, the three composites remained within Jaccard 0.95–1.0 of one another and Spearman rho 0.88–0.95, suggesting that choosing one variant over the other does not have a meaningful effect on the final selection. Full tables and figures for this fixed-corpus robustness check are provided in the 'llm-consolidated-score-analysis' folder of the accompanying data package [23]. This sensitivity analysis targets selection stability and should not be interpreted as external validation of the composite score. Aggregation Sensitivity. In the accompanying mapping study we compared PERT against the mean, median, minimum, and maximum of the three rater scores, holding the threshold at 5.0. The mean and median produced identical selections (Jaccard 1.0, Spearman rho 1.0). PERT, the protocol's recommended choice, selected 55 studies versus 54 for the mean, differing by five additions and four removals (Jaccard 0.848, Spearman rho 0.819). Choosing the minimum rater score was drastically more restrictive, selecting only 12 studies (Jaccard 0.222) with a non-significant rank correlation (rho 0.399, p = 0.199), while the maximum selected 89 studies (Jaccard 0.589). PERT therefore occupies a middle ground closer to the mean and median than to either extreme. Full tables are provided in the 'metrics-analysis' folder of the accompanying data package [23]. Human-only Counterfactual. Because the LLM partition assigns two human raters to every study the LLM evaluates, we could reconstruct a human-only selection by re-aggregating those studies without the LLM and reapplying the threshold. However, we would have to use the simple mean because a PERT-style weighted mean would require three values. Under mean aggregation, removing the LLM changed nine decisions in our own mapping study: seven studies were added and two removed (Jaccard 0.853, Spearman rho 0.691, p < 0.001). Under the minimum statistic the human-only selection was a superset of the LLM-assisted one (two added, none removed), suggesting that human raters assign higher low ratings than LLMs. Using the maximum rater score, the human-only selection was a subset (none added, eight removed) suggesting that LLMs assign higher high ratings. Overall, the LLM's inclusion does not uniformly expand or contract the selection; it both adds and removes studies depending on the aggregation rule. This suggests that using an LLM introduces different biases, rather than acting as a uniformly better or worse rater. Full tables are provided in the 'human-counterfactual' folder of the accompanying data package [23]. Threshold Stability. We examined the sensitivity of the final selection to the quality-assessment threshold using checkpoints at 4.0, 5.0 (the default), and 6.0 under mean aggregation. Raising the threshold only removes studies, never adds them, so the selection is monotone in the threshold. On the selection of our SMS, the retention rate between adjacent checkpoints was approximately 90%: the step from 4.0 to 5.0 removed six studies and the step from 5.0 to 6.0 removed five. The same checkpoints under PERT aggregation produced identical selection sizes (60, 54, and 49 studies), confirming that the default threshold of 5.0 sits in a stable region where a one-step shift in either direction changes the selection by roughly 10% regardless of the aggregation rule. The full sweep across thresholds from 1.0 to 10.0 is provided in the 'threshold-stability' folder of the accompanying data package [23]. Gating Unit. The protocol allows gating the selection at either the category level or the individual quality criterion level. When gating at the category level, a study is included when all four category-level averages meet a threshold. At the criterion level, all eight criterion-level scores must be above the threshold. Under mean aggregation, category-level gating selected 54 studies versus 47 for criterion-level gating, a difference of seven studies that pass the category threshold but fail on at least one criterion within a category. The rank correlation between the two selections was moderate (Spearman rho 0.597, p < 0.001), indicating the two gating units produce differently ordered selections, not just differently sized ones. Under PERT the gap widens slightly to nine studies (55 versus 46, Spearman rho 0.585), but the pattern is the same. Our SMS and all the other analyses presented here used category-level gating because it is more lenient and preserves studies where one criterion is weak but the category as a whole is adequate. Full tables are provided in the 'criteria-dimensions' folder of the accompanying data package [23]. Limitations This method has additional limitations relative to the original SMS method. Internal Generalisation. The method is applicable as-is only to SMS conducted on software architecture. Some data extraction features and quality assessment criteria were formulated with software architecture in mind. In the same vein, the motivation for creating this methodology originated in mapping entity resolution systems, and entity resolution terminology survives in some data extraction features. These limitations may be overcome by adapting the data extraction features and quality assessment criteria on a case-by-case basis. Reference Rater Bias. The reference reviewer (R1) risks introducing systematic errors that propagate unchecked into the final corpus. This is mitigated through a two-stage process applied to both post-search and post-snowballing selection. First, raters were instructed to include when in doubt, deliberately inflating recall. Second, the resulting over-inclusive set was reviewed by the entire team, who reached consensus on exclusions collectively. No study therefore survives into the final corpus on R1′s authority alone; every retained study has been endorsed by the full team, reducing the influence of any single rater's systematic bias. Quality Assessment Selection Bias. The use of quality assessment as an exclusion gate introduces selection bias. Studies that do not meet the minimum quality threshold are removed before data extraction, regardless of whether they describe a relevant system. This reduces the size of the final corpus and may exclude systems that are architecturally significant but poorly reported. The trade-off is intentional: the protocol prioritises actionable architectural knowledge over breadth of coverage. This trade-off is acknowledged as potentially worthwhile in the original SMS methodology. PERT. We acknowledge two limitations. First, the mapping from rater scores to PERT [21] roles is fixed by rank order and is not data-driven. Second, PERT's weighting scheme was designed under assumptions of task-duration uncertainty rather than rater disagreement, and no formal justification exists for its 1:4:1 wt ratio in this context. To allow readers to assess the practical impact of this choice, we report both PERT aggregate scores and median scores in the accompanying data package [23]. Scope. The scope of the study is bounded by the systematic search and snowballing procedures. Systems that lack a peer-reviewed description – regardless of their architectural significance or adoption in practice – fall outside the scope of any SMS conducted using this protocol. This is a deliberate methodological choice, but it means the resulting map does not represent the full landscape of deployed systems. Relaxing it produces an equivalent bias in the opposite direction. A single level of snowballing is suggested when using this method. While this yielded good results for entity resolution, this may be insufficient for research areas where relevant studies are not well-cited by the initial included set. The appropriate number of snowballing levels must be determined on a case-by-case basis. LLM-related Limitations. The LLM-assisted quality assessment component has several technical limitations. PDF-to-text conversion is lossy: formatting, tables, and figures are not faithfully preserved. Text-only extraction is required when not all selected models support vision input. Both factors reduce the information available to the LLM assessor relative to a human reader of the original PDF. Using larger, multi-modal LLMs or reducing the information loss during PDF parsing in other ways requires further examination. The protocol uses zero-shot prompting, which does not provide the LLM with calibration examples. Few-shot prompting may improve agreement with human raters, but it might also contribute to selection bias. The calibration limitation is addressed in part by allowing the LLMs to evaluate only a fifth of the primary studies and by using PERT for producing the final, inter-rater score. LLM behaviour is subject to change over time. Model versions may be deprecated or updated by vendors, altering scoring behaviour without notice. Reproducing the LLM assessment component is possible only using the exact model versions listed in Table 4 to obtain comparable results. Where those versions are no longer available, results may differ even when using the same prompt. Ethics statements Nothing to declare. CRediT author statement Andrei Olar: Conceptualization, Investigation, Writing – Original Draft Oskar Picus: Writing – Review & Editing Laura Cernău: Writing – Review & Editing Laura Dioșan: Writing – Review & Editing, Supervision Simona Motogna: Supervision Data availability The scripts, inputs, and outputs for the sensitivity analyses reported in the Method validation section – LLM model choice, aggregation sensitivity, the human-only counterfactual, threshold stability, and the gating-unit comparison – are openly archived as the Methodology component of the reproduction package of our pilot mapping study on the Open Science Framework [23]. The component is maintained and will be updated to reflect corrections and extensions to the protocol. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.