Meta-analysis compares traditional and modern difference-in-differences estimates

What the study found

In an arXiv preprint, Hull reports an AI-assisted replication analysis of 1,346 staggered-adoption specifications from 108 economics papers. He compares two-way fixed effects (TWFE) estimates with estimates from modern difference-in-differences methods. The comparison addresses concerns about negative weights in TWFE estimation.

Hull reports that the estimates were highly correlated, though their observed signs differed in 10.8–14.5% of comparisons. TWFE estimates were also more precise. Hierarchical models accounting for estimation noise implied a 91–98% correlation between the underlying estimands and a 0.6–2.1% probability of a statistically significant TWFE estimate having the opposite sign from the modern estimand.

Why it matters

Difference-in-differences methods are used to compare changes across groups. In settings where adoption occurs at different times, negative weights in TWFE can complicate interpretation. Hull argues that the results suggest this issue may be less serious in practice than feared. The study also illustrates a possible use of AI-assisted replication for assessing econometric methods at scale.

Limits and unknowns

The findings concern the papers and specifications analyzed, not necessarily every application. The abstract does not identify the papers or modern methods, explain how AI assisted replication, or detail the hierarchical models and their assumptions. It also does not provide enough information to assess why particular estimates differed. The results are reported in a preprint.