Learning the Macroeconomic Language
Siddhartha Chib, Fei Tan · 2025
Read Learning the Macroeconomic Language by Siddhartha Chib, Fei Tan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21031.txt.
Introduction
Siddhartha Chib, Fei Tan, and Zhixun Zhang solve the data famine that blocks language models from macroeconomic forecasting: economies offer decades of quarterly data, not billions of tokens. Their trick is to let economic theory manufacture the corpus. A structural DSGE model is estimated on an initial data segment with stochastic volatility and heavy-tailed shocks. It samples millions of theory-consistent synthetic panels. Those are mixed with the real series to train a transformer.
The structural model acts as a regularizer rather than a forecaster, keeping the network's flexibility inside economically coherent bounds. Against a standard vector autoregression, the hybrid matches or beats predictive performance on token accuracy and likelihood, with the strongest gains under theory-heavy training mixes. Small-sample fields beyond economics can borrow the recipe: simulate from your best theory, then let the network learn the language.
Catalog record
| CREATOR | Siddhartha Chib, Fei Tan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-21031
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning the Macroeconomic Language Siddhartha Chib, Fei Tan Abstract We show how state-of-the-art large language models (LLMs), seemingly inapplicable to the small samples typical of macroeconomics, can be trained effectively for macroeconomic forecasting. We estimate a dynamic stochastic general equilibrium (DSGE) model on an initial segment of the data to obtain a posterior distribution over structural parameters. We sample from this posterior to generate millions of theory-consistent synthetic panels that, when mixed with actual macroeconomic data, form the training corpus for a time-series transformer with attention. The trained model is then used to forecast out-of-sample through 2025. The results show that this hybrid forecaster, which combines the theoretical coherence of DSGE models with the representational power of modern LLMs, learns key features of the macroeconomic language. Keywords : Bayesian methods; Deep learning; DSGE models; Economic forecasting; Large language models; Transformers. JEL Classification : C11, C45, E37 1 Introduction Macroeconomic forecasting has long confronted a fundamental challenge, that of learning reliable dynamic relationships from limited historical data. Quarterly macroeconomic time series rarely exceed a few hundred observations, a sample size that is far too small to support the direct application of modern machine learning methods. In contrast, recent advances in sequence modeling, most notably the transformer architecture of 11 , have been driven by training on billions of examples. The question we address is whether these powerful sequence learners can be adapted to the inherently small-sample environment of macroeconomics. A recent illustration of the flexibility of transformer architectures is provided by the Starcloud-1 space station, where an NVIDIA H100 was used to train the nanoGPT model of 6 on Shakespeare’s complete works, the first large language model (LLM) trained in orbit. This demonstration shows that transformers, once confined to massive data centers, can be deployed under extreme computational constraints through careful architectural design and training strategies. Adapting these methods to macroeconomic forecasting, however, presents a different challenge, that is the scarcity of data rather than computational capacity. In this paper, we show how transformer models can be trained effectively for macroeconomic forecasting by combining two complementary sources of information, limited historical observations and abundant theory-consistent synthetic data that are generated from estimated dynamic stochastic general equilibrium (DSGE) models. The key insight is to use DSGE models not as direct forecasting devices, but as structured data generators that encode economic theory. In this respect, our approach is loosely related to the literature where DSGE models are used to inform priors in reduced-form time-series models [see 2 , 5 , 3 , and 7 ]. Beyond this broad conceptual connection, our method is fundamentally different in its objectives, implementation, and intended application to transformer-based sequence learning. We proceed in two steps. First, we estimate a 9 style DSGE model augmented with stochastic volatility and Student- t shocks using Bayesian methods on an initial segment of historical data. This estimation yields a posterior distribution over structural parameters that captures uncertainty about the economy’s underlying dynamics. We then draw from this posterior to generate millions of synthetic macroeconomic trajectories. Each trajectory represents a plausible counterfactual realization of the economy and incorporates empirically relevant features such as time-varying volatility, fat-tailed shocks, and nonlinear dynamics (Section 2 ). Second, we train a transformer model using batches that combine synthetic observations with real historical data. This mixed training strategy allows the model to absorb economic structure from DSGE-based simulations while remaining anchored to observed macroeconomic outcomes (Section 3 ). The synthetic trajectories expose the transformer to a wide range of economic scenarios, including regimes and crisis episodes that are sparsely represented, or entirely absent, in historical samples. In this way, the procedure directly addresses the small-sample problem that limits the applicability of modern machine learning methods in macroeconomics. Our architectural design incorporates two adaptations that are essential for macroeconomic forecasting. The first concerns the multivariate nature of macroeconomic data. Unlike standard language models, which process a single token at a time, macroeconomic forecasting requires handling several aggregate variables that are observed jointly each period. A naive joint tokenization of all variables would lead to a combinatorial explosion in the vocabulary, rendering the approach infeasible even for modest discretizations. To address this issue, we adopt a separate-then-concatenate embedding strategy in which each variable is tokenized and embedded separately, and the resulting embeddings are concatenated before entering the attention layers of the model. This construction reduces the number of parameters by several orders of magnitude while still allowing the attention mechanism to learn cross-variable dependencies in a flexible manner. The second adaptation concerns the model’s output. Drawing inspiration from modular systems in robotics and autonomous driving, rather than estimating a single, monolithic transformer with a high-dimensional output layer, we employ a modular design consisting of seven specialized transformers, one for each target variable. Each transformer conditions on the same multivariate history but produces forecasts for a single macroeconomic series. This modularization avoids the curse of dimensionality in the output space and yields models that are both stable and computationally efficient. In practice, our largest specification contains on the order of 50,000 parameters and can be trained in minutes on standard hardware, while retaining sufficient capacity to capture nonlinear dynamics and cross-variable interactions. We also depart fundamentally from conventional transfer learning paradigms in which models are pretrained on a large dataset and subsequently fine tuned on a smaller one. In the present setting, such an approach would be ineffective. With only a few hundred quarterly observations relative to the volume of synthetic data, fine-tuning would leave the pretrained representations essentially unchanged, causing the information in the real data to be largely ignored. Instead, we implement mixed-batch training wherein each training iteration samples from both real and synthetic data in a controlled mixture (10% real vs. 90% synthetic in our baseline). This ensures that empirical data influences learning from the outset, while synthetic data stabilizes estimation and mitigates overfitting. The mixing proportion has a interpretation analogous to prior strength in Bayesian inference. Synthetic data encode theoretical restrictions implied by the DSGE model, while real data provide direct empirical evidence. The training process balances these two sources of information in a transparent and controllable manner. More broadly, this approach illustrates how simulation-based models and observational data can be combined within modern deep-learning frameworks, with potential applications beyond macroeconomics to any domain where structural simulators exist but real data are scarce. Our approach connects to a growing body of work in which simulation-based training is used to overcome data scarcity. In autonomous driving, agents are trained extensively in physics-based simulators such as CARLA ( 4 ) before deployment in real environments. In robotics, synthetic manipulation data generated by simulation engines are routinely used to train control policies, with domain randomization techniques ( 10 ) enabling successful transfer from simulated to real-world environments. In each case, the simulator encodes domain-specific structure that allows learning to proceed efficiently despite limited real-world observations. In macroeconomics, DSGE models play an analogous role. They provide theory-consistent simulators that generate economically coherent trajectories, making it possible to train modern transformer-based models even when historical data are scarce. We apply the method to forecast seven key U.S. macroeconomic variables over an out-of-sample period spanning nearly eight years (Section 4 ). The