Reinforcement Learning and Consumption-Savings Behavior
Brandon Kaplowitz · 2025
Read Reinforcement Learning and Consumption-Savings Behavior by Brandon Kaplowitz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20748.txt.
Introduction
Brandon Gary Kaplowitz, writing alone, uses reinforcement learning to explain two odd patterns in how households spend and save during downturns. Standard models assume people form sharp forecasts of future income, yet real spending after job loss and stimulus payments does not fit that neat picture.
In his model, agents learn spending rules from trial and feedback, guided by a neural value estimate whose errors shift with experience. Households with low liquid savings before job loss spend a far larger share of stimulus cash than richer ones, even without borrowing limits. People with more past job loss spells also keep spending lower for longer, a scarring pattern seen in recent studies, and both effects flow from the same learning flaw.
Catalog record
| CREATOR | Brandon Kaplowitz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20748
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reinforcement Learning and Consumption-Savings Behavior Brandon Kaplowitz Abstract This paper demonstrates how reinforcement learning can explain two puzzling empirical patterns in household consumption behavior during economic downturns. I develop a model where agents use Q-learning with neural network approximation to make consumption-savings decisions under income uncertainty, departing from standard rational expectations assumptions. The model replicates two key findings from recent literature: (1) unemployed households with previously low liquid assets exhibit substantially higher marginal propensities to consume (MPCs) out of stimulus transfers compared to high-asset households (0.50 vs 0.34), even when neither group faces borrowing constraints, consistent with [ 9 ] ; and (2) households with more past unemployment experiences maintain persistently lower consumption levels after controlling for current economic conditions, a “scarring” effect documented by [ 25 ] . Unlike existing explanations based on belief updating about income risk or ex-ante heterogeneity, the reinforcement learning mechanism generates both higher MPCs and lower consumption levels simultaneously through value function approximation errors that evolve with experience. Simulation results closely match the empirical estimates, suggesting that adaptive learning through reinforcement learning provides a unifying framework for understanding how past experiences shape current consumption behavior beyond what current economic conditions would predict. 1 Introduction 1.1 Motivation and Goals The overarching goal of this paper is to understand the implications of sophisticated learning protocols for consumption behavior. Specifically, in this project, I focus on two empirical patterns documented in recent years and show that embedding reinforcement learning, a widely used protocol in computer science, in an otherwise standard model of consumption choice can generate these patterns. In my model, a consumer is represented as an economically rational decision-maker with informational and computational constraints. 1 1 1 I refer to this consumer as an agent throughout. In particular, the model assumes that the consumer does not know their income process perfectly, nor are they able to solve a Bellman equation to obtain their true value function. Instead, they learn their continuation value over time using reinforcement learning. They form a parametric estimator of their continuation value and update the parameters of this estimator over time based on their experiences. One crucial assumption that brings considerable tractability is that an agent uses a Markovian formulation of their decision problem, where continuation values depend only on current assets and income. 2 2 2 Other important assumptions made that are standard in an economics context, but may want to be questioned in this context, include: economic rationality of households (that is, that they are subjective expected utility maximizers); agents forming forward-looking expectations; no special memory, context, replay buffer, attention, or model of the environment; choice of learning algorithm among reinforcement learning algorithms; preferences admitting a unique utility representation and this utility representation being available to households; time-separability of preferences; no cost for computation time or other forms of complexity or access to multiple alternative solution schemes; and no preferences regarding the timing of the resolution of uncertainty, such as ambiguity aversion. This is restrictive because it presupposes knowledge of the relevant states, but serves as a natural starting point; with this assumption, the problem becomes defined in terms of a Markov Decision Problem (MDP), where reinforcement learning is an appropriate solution procedure. 3 3 3 Relaxing the agent “magically” knowing their true state variables represents a natural follow-up. See Section A.4 for further details. Two recent empirical findings motivate this analysis. First, [ 9 ] study household spending during the COVID-19 recession and document a puzzling pattern: unemployed households with previously low liquid assets exhibited much higher marginal propensities to consume (MPCs) out of stimulus transfers (0.53) than those with previously high assets (0.29), even when the transfers were large enough that neither group faced current borrowing constraints. Second, [ 25 ] examine how past economic experiences affect current behavior using panel data from the Panel Study of Income Dynamics (PSID) and NielsenIQ Homescan Consumer Panel. They find that households with more past unemployment experiences—either personal or through exposure to high regional unemployment—maintain persistently higher savings rates even after controlling for all current economic conditions including income, wealth, and employment status, and suggest it has a highly persistent effect that they capture via a weighted index. This “scarring” effect suggests that past unemployment experiences create lasting behavioral changes beyond their immediate economic impact. These findings present challenges for standard full-information rational expectation models of consumption, even augmented with borrowing constraints. The model assumes agents approximate their continuation value with a flexible parametric model with a neural network estimate at the rational solution and then proceed to update the parameters governing this model, the weights and biases of the neural network, over the course of their lifetime via gradient descent. To obtain their optimal consumption choice, agents fit a polynomial to the output of the neural network. For initialization, the model assumes each agent begins their life with an estimate of the continuation value that coincides with its rational expectations counterpart. This means that deviations from rational expectations in the model arise from experiences and learning dynamics rather than initial conditions. A key parameter is the learning rate, which controls the rate at which agents update the parameters in response to new observations. The baseline parameterization sets the learning rate to a high initial value and decays it at the rate 1/\sqrt{t} following the work of [ 32 ] to satisfy the criteria of stochastic approximation for stochastic gradient descent and, hence, enable the possibility of convergence in the long run. I simulate the model with standard quarterly consumption-savings parameters heavily drawn from [ 9 , 20 ] . I examine their consumption profiles over 50 quarters. I use the calibrated model to replicate two results that connect to recently documented empirical facts. First, consistent with [ 9 ] , agents who previously held low liquid assets exhibit substantially higher marginal propensities to consume (MPCs) out of stimulus transfers compared to previously high-asset agents, even when transfers of cash are large enough that neither group is currently borrowing constrained. Using data generated via simulation, I find estimates very similar to those found by [ 9 ] . Second, following [ 25 ] , agents with more past unemployment experiences maintain higher savings rates, significant even after controlling for current assets and income and dropping the two most recent observations—a scarring effect from past unemployment experiences. Thus, this paper presents RL as an alternative unifying explanation for these facts. It does not rely on ex-ante heterogeneity—the hypothesis in [ 9 ] —which cannot account for the experience effects observed in [ 25 ] . RL is also distinct from the scarring mechanism in [ 25 ] , as scarring focuses solely on changes in future expected unemployment probabilities. Empirically, my mechanism generates both lower consumption levels and higher MPCs, consistent with the data. In contrast, increased subjective unemployment risk (as learned through scarring) causes MPCs and consumption levels to fall together. This pattern is inconsistent with the higher MPCs that [ 9 ] find among agents with lower endogenous prior assets—consistent with agents who have experienced past unemployment and depleted their savings through consumption-smoothing. 2 Literature This paper is related to several strands of the literature. The substantive question bears a