Experimentation in Gaming: an Adoption Guide

Julian Runge, Draft for feedback · 2025

Read Experimentation in Gaming: an Adoption Guide by Julian Runge, Draft for feedback (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13840.txt.

Introduction

Game studios that test changes on live players grow faster, and a 2025 guide by Julian Runge lays out how to adopt that testing habit. Experimentation runs through the whole life of a game, from first concept checks to live personalization after launch.

Runge structures the guide around the development cycle and the classic marketing mix. Gaming brings its own twists: passionate communities, tangled interactive systems, and player behavior that shifts over time. The guide stresses teamwork across product, marketing, and analytics staff, and it gives practical steps for running fair tests. Ethics get their own treatment, covering fairness and player choice in a medium built on engagement.

Catalog record

CREATORJulian Runge, Draft for feedback
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13840

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Experimentation in Gaming: an Adoption Guide Julian Runge, Draft for feedback Abstract Experimentation is a cornerstone of successful game development and live operations, enabling teams to optimize player engagement, retention, and monetization. This article provides a comprehensive guide to implementing experimentation in gaming, structured around the game development lifecycle and the marketing mix. From pre-launch concept testing and prototyping to post-launch personalization and LiveOps, experimentation plays a pivotal role in driving innovation and adapting game experiences to diverse player preferences. Gaming presents unique challenges, such as highly engaged communities, complex interactive systems, and highly heterogeneous and evolving player behaviors, which require tailored approaches to experimentation. The article emphasizes the importance of collaborative frameworks across product, marketing, and analytics teams and provides practical guidance to game makers how to adopt experimentation successfully. It also addresses ethical considerations like fairness and player autonomy. K eywords experimentation \cdot AB testing \cdot gaming \cdot game development \cdot technology adoption 1 Introduction At the core, in gaming as elsewhere, experimentation and AB testing are about learning from customers, both free and paid, to identify more optimal designs and strategies that drive relevant performance indicators such as engagement, retention, monetization and user satisfaction. A successful experimentation program needs a roadmap for hypothesis-driven learning and innovation, centrally ensured rigor in experiment design and analysis, mechanisms for knowledge accumulation and dissemination [ 1 ] , executive endorsement and a culture that allows for risk-taking and failure [ 2 ] . Gaming, however, offers up unique challenges and opportunities in experimentation. Important differences exist along different stages of the game development and publishing process. Both pre- and post-launch, experimentation informs important steps for bringing a game to market and managing it. While emergent game creation is the focus during earlier stages, commercial optimization becomes more important as a game is live in the market. With game designers, there can be concerns that experiment-driven optimization may compromise the artform that is game design – something that needs to be carefully managed through cross-departmental collaboration and a shared learning agenda. There are further aspects that are unique to experimentation in gaming: Highly engaged player communities can make it hard to run certain types of experiments, e.g., with different prices [ 3 ] , matchmaking algorithms [ 4 ] , or other variations that might be perceived as unfair. As games are highly immersive and interactive experiences, the set of relevant indicators can be larger and more diverse [ 5 ] than in other areas of experimentation. The publishing model (premium versus free-to-play), platform strategy (mobile, console, PC, cross-platform), growth trajectory (paid and/or organic), and ongoing development tactics (LiveOps, content updates) impact to what extent and how you can leverage experimentation. In selecting your game’s technology stack, you should also consider what experimentation and personalization strategies you want to support [ 6 ] once the game is live in market. This article presents an introduction to touchpoints between game making and experimentation, structured against a matrix of a four-stage game development and publishing process and the four Ps of the marketing mix (product, place, price, promotion). To get started, let’s develop this conceptual matrix and populate it with key tasks essential to getting a game to market. We will then discuss how these tasks interact with experimentation, highlight key challenges and opportunities, and outline pathways that will help you succeed with experimentation in gaming. 2 The Game Development and Publishing Process A 2023 blog post by Egor Piskunov [ 7 ] , an experienced game developer, offers up a detailed account with eight essential stages. For our purposes, let’s simplify it a bit to four stages: As noted, experimentation in digital media and interactive entertainment is about learning from consumers, i.e. players. Our conceptualization hence consolidates earlier consumer-distant stages while later in-market stages receive comparatively more attention. To identify the areas where experimentation can drive value at each stage, let’s lean into a concept that captures the strategy set available to companies when interacting with consumers: the marketing mix with its 4P [ 8 ] product, place (aka distribution), price, and promotion. By combining our game lifecycle with the marketing mix, we obtain a matrix that we can populate with key tasks involved in bringing a game idea to market: To get on the same page about what experimentation is: It exposes different versions of a user experience – from button colors over difficulty adaptation systems [ 9 ] and promotional strategies [ 10 ] to whole product features – to customers to learn about their reaction. Pre-launch, experimentation tends to be qualitative and occur on small samples, often without randomization, and with mostly qualitative performance measurement, e.g., through player or expert feedback (is it fun?). During soft launch (i.e., the launch of the game in select test markets), sample sizes grow, randomization becomes a crucial ingredient, and qualitative and quantitative performance measurement tend to both be important. When the game is launched, with live games and large audiences, the emphasis shifts noticeably towards quantitative experimentation with large samples, randomization, and quantitative performance measurement. So now, let’s use our matrix to highlight key tasks that interact significantly with qualitative (so, the pre-launch type with small samples) and quantitative (so, the launch and post-release type with large samples) experimentation, or both. The following matrix will be the backbone for our further discussion: 3 Experimentation Before Launch (Stages 1-3) Working from left to right in our matrix, the pricing and monetization model you choose for your game (1.6) and the intended growth strategy (1.7) will have major impact on your ability to leverage experimentation. Freemium (F2P, free-to-play) games with strong organic growth tend to attract larger audiences, in turn providing a stronger basis and larger sample sizes for experimentation. Monetization design elements such as lootboxes and high-price IAPs (in-app purchases) that skew spending distributions will impact the variance you face in measuring experimental effects. Similarly, if you plan for a paid growth strategy with high-value players and heavy monetization mechanics, you will likely have a (much) smaller player base, impacting your ability to run quantitative experiments. If you also plan to monetize via ads, you may need to run more experiments to optimize overall monetization. If you publish a premium game that players need to buy upfront, you will mostly care about player engagement and retention metrics (and maybe ad monetization). A type of experimentation, sometimes called pre-release experimentation [ 11 ] , often happens in user testing during the game planning and development phases (1.1, 1.2, and 2.2). This type of experimentation is mostly qualitative and hence quite different from experimentation and AB testing on end users and at scale. Sample sizes are much smaller. Examples for pre-release experiments [ 12 ] involve having test users and experts play different prototypes or inviting mock reviews by game journalists and experts. To the extent that generative AI models to emulate in-market player preferences [ 13 ] are available, experiments with prototypes (or later, release candidates in LiveOps, 4.4.) can also be run on AI agents. At a minimum, such experimentation can serve to identify bugs and technical problems [ 14 ] . A typical task in the planning stage that can involve large-scale experiments

Read the complete text