GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis CCS: Software and its engineering Software development process management CCS: Applied computing Enterprise computing

Alex Heilman, Alex Kyllo, Emerson Murphy-Hill · 2026

Read GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis CCS: Software and its engineering Software development process management CCS: Applied computing Enterprise computing by Alex Heilman, Alex Kyllo, Emerson Murphy-Hill (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00438.txt.

Introduction

Heilman, Kyllo, and Murphy-Hill weigh if GitHub Copilot lifts coder yield, dose by dose. Heavy users may just be strong coders to start with. Busy weeks may also swell both tool use and done work. The team use 43 weeks of data on more than 16,000 coders at Microsoft.

Each coder is set next to their own past self in the math. That kills fixed gaps in skill, role, and team. Code time and browse time enter the fit to soak up busy-week swell. Done pull requests per set work time form the yield mark. Top-use weeks show far more done pulls than no-use weeks. The climb is smooth, with gains that taper at high dose. Seven stress tests leave the tale in place.

Catalog record

CREATORAlex Heilman, Alex Kyllo, Emerson Murphy-Hill
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.00438

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GitHub Copilot and Developer Productivity: An Observational Dose-Response Analysis CCS: Software and its engineering Software development process management CCS: Applied computing Enterprise computing Alex Heilman, Alex Kyllo, Emerson Murphy-Hill Abstract . Does GitHub Copilot (GHCP) make engineers more productive, or do the engineers who use it more differ from those who use it less? And even within a single engineer, are GHCP-heavy weeks just busy weeks in which more of everything gets done? We study these questions using 43 weeks of data from 16,223 software engineers across Microsoft’s Cloud+AI organization. Engineer fixed effects address the first concern by comparing each engineer against themselves rather than against other engineers, eliminating time-invariant differences in skill, role, and team. Active coding time and browser time then enter a Poisson Pseudo-Maximum Likelihood model with two-way fixed effects to address the harder, within-engineer confound: that GHCP-heavy weeks coincide with high-effort weeks. This defines our estimand as an efficiency effect: more pull requests completed at equivalent levels of coding time. Engineers are estimated to complete 40.5% more PRs in their highest GHCP usage weeks relative to their zero-usage weeks, holding measured development effort constant. The gradient is monotonic with diminishing returns at high intensity. Seven robustness and falsification tests target the remaining plausible alternative explanations (non-coding AI engagement, team-level shocks, within-week task reallocation, cross-week contamination, PR slicing into smaller units, shifts toward easier task types, and sensitivity to how the treatment is operationalized). Under an explicitly stated conditional-independence assumption, the within-engineer design estimates a tool-specific efficiency effect that is consistent with all seven robustness tests. 1. Introduction & Motivation Measuring the productivity impact of LLM coding assistants is deceptively difficult. A growing empirical literature examines how Copilot affects developer work, including controlled experiments by Peng et al. ( Peng et al. 2023 ) , quasi-experiments on task allocation by Hoffmann et al. ( Hoffmann et al. 2025 ) , and randomized field experiments by Cui et al. ( Cui et al. 2025 ) reporting productivity gains. Extending these findings from initial adoption to the steady-state productivity of a ubiquitous tool requires a new methodology. Why not an A/B test? The gold standard for causal inference is not feasible here. GHCP is already widely deployed, and removing an established tool from a random subset of engineers raises practical and ethical concerns. This would disrupt workflows, drive engineers to an alternative, and measure the cost of losing a tool rather than the benefit of having one. One could A/B test an incremental GHCP feature (e.g., a new model version), but that only measures the marginal improvement, not the overall productivity impact of GHCP. 1.1. Two Layers of Confounding The motivating question hides two distinct confounds, and our design addresses them in two stages. Between-engineer selection. The engineers who use GHCP more may differ systematically from those who use it less—more skilled, on different teams, working on PR-friendly task mixes, or simply more receptive to new tools. Any cross-sectional comparison conflates these differences with the tool’s effect. Within-engineer, time-varying confounding. Even for a fixed engineer, the weeks in which they use GHCP heavily may not be comparable to the weeks in which they use it lightly. A heads-down crunch week, a large feature landing, or a productive stretch could plausibly raise both GHCP usage and PR output for reasons unrelated to the tool’s marginal contribution. 1.2. Approach: Within-Engineer Dose-Response This analysis exploits within-engineer, week-to-week variation in GHCP usage intensity to address both layers. Engineer fixed effects fully absorb the time-invariant between-engineer selection problem. We compare each engineer against themselves, not against other engineers, so any time-invariant difference in ability, role, team, or task allocation is differenced out. Active coding time and browser time then address the harder within-engineer confound by holding measured development effort constant in the same week. Week fixed effects absorb shocks shared across the population. Assumption 1 (Conditional independence). Conditional on engineer fixed effects, week fixed effects, coding time, and browser time, an engineer’s GHCP usage intensity is not systematically driven by the same unobserved factors that independently determine their PR output that week. Like any non-experimental identification, this assumption cannot be tested directly. Section 4 probes its plausibility through a seven-test falsification battery, and Section 5 characterizes the threats it cannot rule out. 1.3. Causal Inference Context Pearl’s “Ladder of Causation” distinguishes three levels of causal reasoning, each requiring progressively stronger evidence ( Pearl and Mackenzie 2018 ) . (1) Association: observing that GHCP users complete more PRs. (2) Intervention: what would happen if we made an engineer use GHCP more? Only a randomized experiment answers this directly. (3) Counterfactual: what would have happened to this specific engineer in this specific week had they not used GHCP? A randomized A/B test would place us on rung 2. That is not feasible here for the reasons described above. We therefore operate on rung 1, but with design choices that narrow the gap toward rung 2: • Within-engineer comparison. Engineer fixed effects eliminate all time-invariant confounds (ability, experience, team, role, coding style). We compare the same person across weeks, not different people. • Calendar fixed effects. Week fixed effects absorb time-varying shocks that are common to the population (deadlines, holidays, reorgs). • Effort controls as estimand choice. Coding time and browser time enter the model as controls, which defines the estimand as an efficiency effect: the association between GHCP usage and PR output, holding measured development effort constant. If GHCP reduces the time needed to complete a task, that channel is absorbed by the controls, making our estimates conservative (see Section 2.4 ). • Falsification battery. Sections 4.1 – 4.7 systematically test whether the remaining time-varying confounds can explain the gradient. 2. Methodology 2.1. The Confounded Baseline Before introducing the specification, we establish the naive comparison as motivation. The simplest question one might ask is whether engineers who use GitHub Copilot more complete more PRs, and at the raw weekly average level the answer is yes. That answer is not informative, however, because engineers who use GHCP more also tend to spend more time coding in those weeks. Any positive cross-sectional association between usage and PR output is mechanically confounded by effort, and we cannot tell from such a comparison whether GHCP is contributing to productivity or whether we are merely observing a correlation with hours worked. This motivates the specification developed in the remainder of the section. 2.2. Fixed-Rate Offset Model A natural starting point is to model productivity as a rate: completed PRs per active coding hour. We accomplish this by placing \ln(\text{CodingTime}) as an offset in a Poisson regression: \displaystyle\ln E\!\left[\frac{\text{PRs}_{it}}{\text{CodingTime}_{it}}\right]=\beta_{1}\cdot\text{Low}_{it}+\beta_{2}\cdot\text{Mod}_{it} \displaystyle+\beta_{3}\cdot\text{High}_{it}+\delta_{t}+\alpha_{i} This forces a proportionality assumption, in which doubling coding hours must double the PRs. Each \beta then measures the percentage change in the PR rate at a given GHCP usage level relative to no usage, neatly isolating an efficiency effect if the assumption holds. The assumption is empirically false in our panel. Figure 1 plots average PRs against active coding hours alongside the proportional reference line the offset spec assumes. The observed curve falls below that line and flattens as hours rise, consistent with diminishing returns to effort. Because GHCP usage and coding hours are themselves correlated in the panel, any apparent

Read the complete text