Life Sequence Transformer: Generative Modelling for Counterfactual Simulation
Alberto Cabezas, Carlotta Montorsi · 2025
Read Life Sequence Transformer: Generative Modelling for Counterfactual Simulation by Alberto Cabezas, Carlotta Montorsi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01874.txt.
Introduction
Alberto Cabezas and Carlotta Montorsi present a 2025 paper on simulating life courses with transformers. The head title frames it as generative modelling of socio-economic trajectories from administrative data. Lives unfold as sequences of jobs, benefits, and family events. The team encodes those histories as sequences, with overlapping events across life domains kept intact. A transformer learns the patterns at scale. Training data come from the Italian social security administration.
The trained model reproduces known labor market patterns. It also generates coherent what-if life paths given a person's past. Policy teams get a simulator for counterfactual lives: what if a reform, a layoff wave, or a training program hit at a given age? Such paths are hard to test ethically in real life. A generative twin of the registry offers a sandbox, with realism checks against known causal facts.
Catalog record
| CREATOR | Alberto Cabezas, Carlotta Montorsi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01874
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Life Sequence Transformer: Generative Modelling for Counterfactual Simulation Alberto Cabezas, Carlotta Montorsi Abstract Social sciences rely on counterfactual analysis using surveys and administrative data, generally depending on strong assumptions or the existence of suitable control groups, to evaluate policy interventions and estimate causal effects. We propose a novel approach that leverages the Transformer architecture to simulate counterfactual life trajectories from large-scale administrative records. Our contributions are: the design of a novel encoding method that transforms longitudinal administrative data to sequences and the proposal of a generative model tailored to life sequences with overlapping events across life domains. We test our method using data from the Istituto Nazionale di Previdenza Sociale (INPS), showing that it enables the realistic and coherent generation of life trajectories. This framework offers a scalable alternative to classical counterfactual identification strategy—such as difference-in-differences and synthetic controls—particularly in contexts where these methods are infeasible or their assumptions unverifiable. We validate the model’s utility by comparing generated life trajectories against established findings from causal studies, demonstrating its potential to enrich labour market research and policy evaluation through individual-level simulations. 1 Introduction Counterfactual inference is a cornerstone of scientific reasoning in disciplines such as political science Sekhon 2009 , economics ( Heckman 2008 ) , history ( Levy 2015 ) , and psychology ( Foster 2010 ) , and has also sparked ongoing methodological debate ( Cartwright 2007a ) . Counterfactual identification requires constructing hypothetical scenarios that resemble what would have happened in the absence of an event assumed to be causal ( Rubin 2005 ) . Such a scenario allows quantifying the impacts of specific events—like treatments or policies—or the underlying causes of observed outcomes. Typical research questions include: What is the causal effect of increasing the minimum wage on household consumption ( Alonso 2022 ) ? What are the drivers of civil conflict ( Acemoglu et al. 2020 ) ? The gold standard approach for identifying causal effects involves randomised interventions, in which the treatment is assigned randomly to some individuals and withheld from others ( Cartwright 2007b ) . The difference in outcomes between the treated and control groups is interpreted as the causal effect of treatment. Randomisation ensures that characteristics that might correlate with the outcome of interest, such as gender, age, and socioeconomic status, are, on average, evenly distributed across the treatment and control groups. Thus, any systematic outcome difference can be attributed to the treatment itself. However, randomisation rarely occurs naturally and can be challenging to implement even in controlled settings, due to ethical constraints and substantial costs ( Thomas 2016 ) . As a result, economists and other researchers have developed strategies and mathematical models to identify suitable control groups without relying on random treatment assignment. Despite being widely adopted, these observational methods often rely on specific assumptions about the data and the underlying causal mechanisms, which are typically difficult to verify in practice (for instance, see Appendix A ). Our contributions This paper addresses a fundamental challenge in counterfactual inference: how to construct hypothetical scenarios that are both realistic and practically obtainable, especially when randomisation is unfeasible. We introduce a novel encoding method for life trajectories derived from administrative data in Section 3 . Then, detail a Transformer foundational model for the generative task in Section 4 . The model is trained on 1.6 million individual labour trajectories from INPS data, enabling scalable simulation of counterfactual life paths. In Section 5 , our empirical analysis shows the generative potential of our encoding and architecture through two benchmarking exercises: evaluating its ability to simulate realistic life sequences aligned with traditional causal inference-driven findings in Section 5.2 and evaluating the coherence of generated out-of-sample trajectories in Section 5.3 . Once trained, the model can generate realistic alternative life sequences by injecting or removing specific events from an individual’s observed trajectory, thus constructing counterfactuals without relying on external control groups. 2 Background Our work builds on recent advances in Transformer-based sequence modelling for longitudinal labour and health data. In particularly, Life2Vec ( Savcisens et al. 2024 ) , CAREER ( Vafa et al. 2024 ) , and TransformEHR ( Yang et al. 2023 ) . Life2Vec showed the validity of encoder-only Transformer models for modelling socio-economic data and outperformed the predictive performance of standard models—like mortality tables, logistic regression, or recurrent neural networks—on individual’s future life outcomes (e.g., mortality) or time-invariant characteristics (e.g., personality nuances). However, their model lacks generative capability to simulate sequences of future tokens, and their data encoding cannot be easily adapted to a generative task. In contrast, our proposed encoding approach enables the training of a generative decoder, allowing for the simulation of counterfactual life sequences on a model architecture directly inspired by Life2Vec. The CAREER encoder-decoder Transformer model focused on job sequence generation with annual temporal resolution and no rich occupational attributes, such as work province or work title. We enhance this framework by introducing fine-grained temporal encoding and a richer set of characteristics that span labour and demographic dimensions. In CAREER, time is introduced with a timestep of one year, and there is no way of modelling the duration of events or the exact timing of occurrence. You could increase the timestep to monthly steps, but this would restrict generation to a fixed monthly grid, offering no control over when within the year events begin or how long they last. Our method of encoding time, by contrast, enables flexible temporal generation by allowing each life event to have a defined starting month and duration in months within the year. TransformEHR is an encoder-decoder Transformer generative model which encodes sequences of electronic health records and predicts the subsequent closest diagnoses (ICD-10 code). It includes demographic and time embeddings to model the numerical format of the visit date. TransformEHR was pre-trained to predict a patient’s future visits, complete ICD codes, given longitudinal ICD codes up to the current visit. Similarly to Life2Vec, the timing of each visit must be specified at inference time—meaning the model cannot decide when events occur, but only generate diagnoses for predefined dates. In contrast, our model allows the generation process to determine when events occur and the duration of each event. Crucially, our model supports variable event durations, which is essential for modelling life sequences that extend beyond instantaneous diagnoses. Another relevant work is the REMI data encoding ( Huang and Yang 2020 ) within the Figaro encoder-decoder Transformer ( von Rütte et al. 2023 ) , which models sequences in the musical domain through structurally rich encodings. The encoding approach informs our design of temporally structured representations, which we adapt to model life events rather than notes. REMI encodes rhythmic structure using the beat-bar-phrase hierarchical structure in music into the sequence. Specifically, encoding a <Bar> event to indicate the beginning of a bar and the <Pos> events to point to certain locations within a bar. We use these representations in our encoding to represent the years and months when a given event occurs. Finally, recent work on causal representation learning (e.g., Vafa et al. 2025 , Gloeckle et al. 2024 ) explores debiased fine-tuning and interpretable interventions; however, it does not explore generative, individual-level counterfactuals. Our approach fills this gap by combining causal reasoning with