Improving Generative Ad Text on Facebook using Reinforcement Learning
Daniel R. Jiang, Alex Nikulkov, Yu-Chia Chen, Yang Bai, Zheqing ZhuMeta Platforms, Menlo Park, California, USA · 2025
Read Improving Generative Ad Text on Facebook using Reinforcement Learning by Daniel R. Jiang, Alex Nikulkov, Yu-Chia Chen, Yang Bai, Zheqing ZhuMeta Platforms, Menlo Park, California, USA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21983.txt.
Introduction
Five authors tune ad text with reward learning on Facebook in a 2025 study of a live shop tool. The tool, AdLlama, spins fresh takes on hand ad text for shops of all sizes. A post-train step pays the model with past ad score marks, not hand picks from a set list. The step is named for that pay path in the work.
A ten-week A/B test spans near 35,000 shops and 640,000 ad takes in the live run. Click rates rose 6.7 percent over a taught copy build in the test, with firm marks. Shops that used the tool made more takes per shop, a joy mark in the read. Gains held across the full test span.
Catalog record
| CREATOR | Daniel R. Jiang, Alex Nikulkov, Yu-Chia Chen, Yang Bai, Zheqing ZhuMeta Platforms, Menlo Park, California, USA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Improving Generative Ad Text on Facebook using Reinforcement Learning Daniel R. Jiang, Alex Nikulkov, Yu-Chia Chen, Yang Bai, Zheqing ZhuMeta Platforms, Menlo Park, California, USA Abstract Generative artificial intelligence (AI), in particular large language models (LLMs), is poised to drive transformative economic change. LLMs are pre-trained on vast text data to learn general language patterns, but a subsequent post-training phase is critical to align them for specific real-world tasks. Reinforcement learning (RL) is the leading post-training technique, yet its economic impact remains largely underexplored and unquantified. We examine this question through the lens of the first deployment of an RL-trained LLM for generative advertising on Facebook. Integrated into Meta’s Text Generation feature, our model, “AdLlama,” powers an AI tool that helps advertisers create new variations of human-written ad text. To train this model, we introduce reinforcement learning with performance feedback (RLPF), a post-training method that uses historical ad performance data as a reward signal. In a large-scale 10-week A/B test on Facebook spanning nearly 35,000 advertisers and 640,000 ad variations, we find that AdLlama improves click-through rates by 6.7% ( p = 0.0296 p=0.0296 ) compared to a supervised imitation model trained on curated ads. This represents a substantial improvement in advertiser return on investment on Facebook. We also find that advertisers who used AdLlama generated more ad variations, indicating higher satisfaction with the model’s outputs. To our knowledge, this is the largest study to date on the use of generative AI in an ecologically valid setting, offering an important data point quantifying the tangible impact of RL post-training. Furthermore, the results show that RLPF is a promising and generalizable approach for metric-driven post-training that bridges the gap between highly capable language models and tangible outcomes. 1 Introduction Generative artificial intelligence (AI) is increasingly being recognized for its transformative potential across industries, driving innovations in content creation [ 1 , 2 , 3 , 4 , 5 ] , education [ 6 , 7 , 8 ] , medicine [ 9 , 10 , 11 , 12 ] , and complex decision-making [ 13 , 14 , 15 ] . It is also widely believed to have the potential for significant economic impact [ 16 , 17 , 18 , 19 , 20 ] , with pioneering work showing quantifiable benefits in the areas of customer support [ 21 , 22 ] , enterprise information tasks [ 23 ] , legal tasks [ 24 , 25 ] , consulting work [ 26 ] , software development [ 27 , 28 , 18 ] , email marketing [ 29 ] , online advertising [ 30 ] , and a variety of professional writing tasks [ 31 ] . However, turning this potential into consistent real-world impact depends critically on how models are fine-tuned and aligned during a phase of model training called post-training . The initial phase for large language models (LLM) training is called pre-training , where the model learns general language patterns and world knowledge from large-scale unlabeled text data. However, to perform effectively in real-world applications, generative AI systems require some form of post-training —a suite of methods designed to adapt pre-trained foundation models to specific downstream tasks [ 32 ] . A common approach involves supervised fine-tuning (SFT), in which models are trained to follow human instructions by mimicking curated target responses [ 33 , 34 ] . Another example is the use of reinforcement learning from human feedback (RLHF) to train the model on human preference data [ 35 ] , which has been shown to be highly effective in domains like summarization [ 36 ] , chatbot assistants [ 37 ] , and instruction-following [ 38 ] . More recently, researchers have found that reinforcement learning (RL) using verifiable rewards (i.e., objective signals from domains like math or coding that can be automatically validated) can lead to marked improvements in LLM reasoning and problem-solving capabilities [ 32 , 39 ] . As noted above, there is a broadening body of research focused on quantifying the impact of LLMs across various sectors. However, the specific impact of the post-training phase has been relatively underexplored, despite its critical role in the development of virtually all prominent LLM models. In this paper, we present new findings on the concrete impact of RL for post-training. Our analysis is conducted through the perspective of the online advertising industry, which is a major driver of the global economy: worldwide expenditure in online ads is projected to reach $513 billion USD in 2025 [ 40 ] and is expected to represent 63% of the total global advertising revenue across all mediums [ 41 ] . Specifically, we focus on Meta’s Text Generation product, which uses an LLM to generate variations of an advertiser’s human-written ad text. This allows each advertiser to select multiple versions of the ad to show to potential customers, taking advantage of Meta’s ad delivery systems to show the most performant variants. The initial version of the Text Generation product used an LLM that was trained in a naive manner, by fine-tuning it to imitate the style of a set of curated ads using supervised learning (SFT). Figure 1 : Overview of our contributions. Our first contribution, illustrated in the left panel, is RLPF, a reinforcement learning (RL) approach to post-training an LLM based on an aggregate performance metric. We apply RLPF to a generative AI feature in Meta’s Ads Manager that helps advertisers generate new ad text variations. To do this, we use Facebook ad performance data (i.e., click-through rates) to train a reward model, which can score the effectiveness of a piece of ad text. Subsequently, we align the LLM toward this reward model using RL, which involves iteratively generating and scoring ad text, illustrated in lower part of the figure. The resulting LLM is called “AdLlama.” Our second contribution, illustrated in the right panel, are the results of a large-scale advertiser A/B test, encompassing approximately 35,000 advertisers and 640,000 ad variations, which showed that advertisers who received AdLlama achieved 6.7% higher advertiser-level click-through rates (CTR). To our knowledge, this study is the largest reported so far that investigates the use of generative AI in an ecologically valid setting. In this paper, we take the next step and describe our efforts in using reinforcement learning to improve the Text Generation LLM, with the measurable goal of writing more engaging ad text. We do this by directly using performance (i.e., click-through rates) as a reward signal. We call the approach reinforcement learning with performance feedback (RLPF). By viewing each user’s behavior (i.e., click or no click) on each ad impression as a small piece of human feedback, RLPF can be thought of as an extension of the well-known RLHF technique [ 35 , 33 ] , except that each piece of ad text is being “rated” by thousands of humans (i.e., Facebook users who see an ad impression) via click or no click signals. In that sense, one can view RLPF as falling somewhere in between traditional RLHF and the recently explored direction of RL with verifiable rewards [ 32 ] . We report on the results of a large-scale online experiment (i.e., A/B test) conducted on Facebook