Słoczyński clarifies when the OLS coefficient on a treatment dummy can and cannot be read as the ATE.
Consider the standard setup:
\[ y = \alpha + \tau d + X \beta \tag{12.1}\]
Here \(d\) is the treatment dummy and \(X\) the covariates, under unconfoundedness. If the treatment effect is homogeneous, \(\tau\) is the ATE. Under heterogeneous effects, \(\tau\) is a convex combination of ATT and ATU — but with weights that are inversely proportional to the treatment shares. The more treated units there are, the less weight falls on ATT.
where \(\rho = P(d=1)\), the proportion of treated; \(\tau_{ATT} = E(y(1)-y(0) | d=1)\), \(\tau_{ATU} = E(y(1)-y(0) | d=0)\), and \(\tau_{ATE} = E(y(1)-y(0))\).
From these two equations, we see that \(\tau\) and \(\tau_{ATE}\) are quite different, unless the weight on ATT happens to equal \(\rho\). A caution on reading this too simply: the clean form \(\tau = (1-\rho)\tau_{ATT} + \rho\tau_{ATU}\) with \(\rho = P(d=1)\) is the no-covariate special case (or, with covariates, holds in terms of covariate-adjusted treatment-probability weights, not the raw marginal share). In Słoczyński’s general result the weights depend on the conditional treatment probabilities (the propensity score), so a 50/50 marginal treatment share alone does not guarantee that OLS recovers the ATE — what matters is the covariate-adjusted weighting. With that caveat, the intuition stands: the more imbalanced the (conditional) treatment probabilities, the further OLS can be from the ATE.
Given the true propensity score, ATT, ATU, and ATE can be separated. In practice we use the estimated propensity score. Słoczyński provides the hettreatreg package in R and Stata.
The data are nswcps: the NSW experimental treatment group combined with a large CPS comparison sample, so the treated are a tiny fraction of the whole. The outcome is 1978 earnings and the covariates are age, its square, education, race, marital status, degree status, and earnings in 1974 and 1975. This is the classic LaLonde setting, and its extreme treatment imbalance is exactly what makes the weighting result bite.
Code
library(hettreatreg)data("nswcps")summary(lm(re78~treated+age+age2+educ+black+hispanic+married+nodegree+re74+re75, data =nswcps))
Call:
lm(formula = re78 ~ treated + age + age2 + educ + black + hispanic +
married + nodegree + re74 + re75, data = nswcps)
Residuals:
Min 1Q Median 3Q Max
-25130 -3601 1274 3668 55040
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 7634.34415 736.67074 10.363 < 2e-16 ***
treated 793.58704 548.25433 1.447 0.14778
age -233.67749 41.18067 -5.674 1.42e-08 ***
age2 1.81437 0.56099 3.234 0.00122 **
educ 166.84923 28.65984 5.822 5.94e-09 ***
black -790.60856 213.24523 -3.708 0.00021 ***
hispanic -175.97512 218.99126 -0.804 0.42166
married 224.26599 149.84542 1.497 0.13450
nodegree 311.84453 178.51743 1.747 0.08068 .
re74 0.29534 0.01222 24.175 < 2e-16 ***
re75 0.47064 0.01216 38.700 < 2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 7002 on 16166 degrees of freedom
Multiple R-squared: 0.4762, Adjusted R-squared: 0.4758
F-statistic: 1469 on 10 and 16166 DF, p-value: < 2.2e-16
"OLS" is the estimated regression coefficient on treated.
OLS = 793.6
P(d=1) = 0.011
P(d=0) = 0.989
w1 = 0.983
w0 = 0.017
delta = -0.971
ATE = -6751
ATT = 928.4
ATU = -6840
OLS = w1*ATT + w0*ATU = 793.6
The output is the theorem in numbers. OLS puts 793.6 on treated. Only 1.1% of the sample is treated, \(P(d=1) = 0.011\), and the implied OLS weights are \(w_1 = 0.983\) on ATT and \(w_0 = 0.017\) on ATU. Compare those with the ATE’s own weights, which are \(\rho = 0.011\) on ATT and \(0.989\) on ATU: they are very nearly reversed. The package prints the identity directly, \(\text{OLS} = w_1\,\text{ATT} + w_0\,\text{ATU} = 793.6\), which closes exactly.
The consequences are not subtle. ATT is 928.4, ATU is \(-6840\), and ATE is \(-6751\). So the OLS coefficient of 793.6 sits within 135 of the ATT and roughly 7,500 away from the ATE — and on the opposite side of zero. Reading 793.6 as “the effect of training on earnings” would get the sign wrong for the population. The delta of \(-0.971\) that the package reports is the gap between the OLS weight on ATT and \(\rho\); it is close to \(-1\), its extreme value.
Słoczyński’s paper itself is an algebraic decomposition of the OLS coefficient \(\tau\) in terms of ATT/ATU weights; it does not propose or endorse the propensity-score-as-covariate estimator below as a general-purpose way to recover ATT/ATU/ATE. What follows is our own illustrative exercise, done in three steps, to see how such an estimator would behave in practice: First, estimate the propensity score equation, basically
\[ d = X \gamma \tag{12.4}\]
Second, include the predicted propensity score as a covariate and estimate the model with treatment and control group separately. Note that regressing \(Y\) linearly on the scalar propensity score \(p(X)\) assumes \(E[Y(d) \mid p(X)]\) is linear in \(p(X)\) — a strong and generally untested assumption, unlike matching/weighting directly on the propensity score, or regression adjustment on the covariates \(X\) themselves, which are the standard approaches. Treat the numbers below as an exploratory illustration rather than a recommended estimator.
Third, predict the counterfactuals for the full sample. Then we get the ATE, ATT and ATU.
One more reason to treat this as illustration rather than method: the propensity score here comes from a linear probability model, and on these data it runs from \(-0.034\) to \(0.154\) — some fitted “probabilities” are negative. That is harmless when the score is used merely as a regressor, as it is here, but it would be fatal if the same score were used as an inverse-probability weight.
Let’s see with the same data:
Code
m_propensity<-lm(treated~age+age2+educ+black+hispanic+married+nodegree+re74+re75, data=nswcps)ps<-predict(m_propensity)df_combined<-data.frame(nswcps, ps)df.1<-df_combined[which(treated==1),]df.0<-df_combined[which(treated==0),]m_ot<-lm(re78~ps, data =df.1)ot<-suppressWarnings(predict(m_ot, newdata =df_combined))# add newdata option to predict for full samplem_oc<-lm(re78~ps, data =df.0)oc<-suppressWarnings(predict(m_oc, newdata =df_combined))# add newdata option to predict for full samplete<-ot-ocate<-mean(te)df_combined$te<-teatt<-as.numeric(mean(df_combined[which(treated==1),'te']))atu<-as.numeric(mean(df_combined[which(treated==0),'te']))print(paste("ate=", signif(ate, 4)))
This reproduces the package’s decomposition exactly: ATE \(-6751\), ATT 928.4, ATU \(-6840\), the same three numbers to four significant figures.
We see in this case OLS coefficient is hugely different from ATE. This is because \(P(d=1)\) is only .01, about 1 percent; we have much larger control group than treatment group. In this case, the OLS coefficient is far different from ATE; in fact, it’s pretty close to ATT.
Now, what’s the intuition that OLS coefficient is just the opposite as the ATE in terms of weighting ATT and ATU? The reason is that OLS is to predict actual outcome, not counterfactual outcome. For calculating ATE, we need counterfactual outcome predicted. Now suppose we have a lot of control observations, we’d much better predicting \((y(0)| d==0)\) rather than \((y(1) | d==1)\). That is, we are better getting the slope of \((y(0) | d==0, X=x)\); therefore predicting \((y(0) | d==1, X=x)\). Therefore although treatment group has less observations, but because the \((y(0) | d==1)\) is better predicted, the ATT is better estimated, thus more weight. If the opposite happens, then the ATU should be more heavily weighted. Thus the counter-intuitive weighting.
12.2 Regression adjustment
Słoczyński’s method is to include propensity score as a covariate. This propensity score serves as a proxy as all covariates. As Rosenbaum and Rubin (1983) showed, the propensity score is a balancing score (treatment is independent of the covariates given the propensity score). Another way to allow treatment effect to differ across all covariates is to allow interaction of treatment dummy with all covariates, then I am doing “Regression Adjustment” (see Stata’s teffects ra, used in the treatment effects and matching chapter; note this is not treatreg, which was the old endogenous-treatment command, superseded by etregress). We can calculate ATE, ATT, ATU after the linear model.
Regression adjustment gives ATE \(-4930\), ATT 796 and ATU \(-4996\).
We see this is somewhat different from Słoczyński’s method, which gave \(-6751\), 928.4 and \(-6840\): the two agree on sign and order of magnitude but not on level, unsurprisingly, since one regresses on a scalar propensity score and the other interacts the treatment with all nine covariates. But the point is the original OLS coefficient 794 is nowhere close to ATE under either approach — \(-4930\) or \(-6751\), both far away and both negative. In this case, it’s very close to ATT (796 by regression adjustment, 928 by the propensity-score route), as the treatment group is very small.
---title: "When is OLS coefficient the ATE"date: "2021-11-03"---## OLS cannot be ATE (most of the time)[Słoczyński](https://doi.org/10.1162/rest_a_01072) clarifies when the OLS coefficient on a treatment dummy can and cannot be read as the ATE.Consider the standard setup:$$ y = \alpha + \tau d + X \beta $$ {#eq-ols-ate-1}Here $d$ is the treatment dummy and $X$ the covariates, under unconfoundedness. If the treatment effect is homogeneous, $\tau$ is the ATE. Under heterogeneous effects, $\tau$ is a convex combination of ATT and ATU — but with weights that are inversely proportional to the treatment shares. The more treated units there are, the less weight falls on ATT.By definition,$$ \tau_{ATE} = \rho \tau_{ATT} + (1- \rho) \tau_{ATU} $$ {#eq-ols-ate-2}The main result from the paper is:$$ \tau = (1- \rho) \tau_{ATT} + \rho \tau_{ATU} $$ {#eq-ols-ate-3}where $\rho = P(d=1)$, the proportion of treated; $\tau_{ATT} = E(y(1)-y(0) | d=1)$, $\tau_{ATU} = E(y(1)-y(0) | d=0)$, and $\tau_{ATE} = E(y(1)-y(0))$.From these two equations, we see that $\tau$ and $\tau_{ATE}$ are quite different, unless the weight on ATT happens to equal $\rho$. A caution on reading this too simply: the clean form $\tau = (1-\rho)\tau_{ATT} + \rho\tau_{ATU}$ with $\rho = P(d=1)$ is the **no-covariate special case** (or, with covariates, holds in terms of *covariate-adjusted* treatment-probability weights, not the raw marginal share). In Słoczyński's general result the weights depend on the conditional treatment probabilities (the propensity score), so a 50/50 *marginal* treatment share alone does **not** guarantee that OLS recovers the ATE — what matters is the covariate-adjusted weighting. With that caveat, the intuition stands: the more imbalanced the (conditional) treatment probabilities, the further OLS can be from the ATE.Given the true propensity score, ATT, ATU, and ATE can be separated. In practice we use the estimated propensity score. Słoczyński provides the `hettreatreg` package in R and Stata.The data are `nswcps`: the NSW experimental treatment group combined with a largeCPS comparison sample, so the treated are a tiny fraction of the whole. Theoutcome is 1978 earnings and the covariates are age, its square, education, race,marital status, degree status, and earnings in 1974 and 1975. This is the classicLaLonde setting, and its extreme treatment imbalance is exactly what makes theweighting result bite.```{r}#| label: chunk1library(hettreatreg)data("nswcps")summary(lm(re78 ~ treated + age + age2 + educ + black + hispanic + married + nodegree + re74 + re75, data = nswcps))outcome <- nswcps$re78treated <- nswcps$treatedour_vars <-c("age", "age2", "educ", "black", "hispanic", "married", "nodegree", "re74", "re75")covariates <-subset(nswcps, select = our_vars)hettreatreg(outcome, treated, covariates, verbose =TRUE)```The output is the theorem in numbers. OLS puts 793.6 on `treated`. Only 1.1% ofthe sample is treated, $P(d=1) = 0.011$, and the implied OLS weights are$w_1 = 0.983$ on ATT and $w_0 = 0.017$ on ATU. Compare those with the ATE's ownweights, which are $\rho = 0.011$ on ATT and $0.989$ on ATU: they are very nearlyreversed. The package prints the identity directly,$\text{OLS} = w_1\,\text{ATT} + w_0\,\text{ATU} = 793.6$, which closes exactly.The consequences are not subtle. ATT is 928.4, ATU is $-6840$, and ATE is$-6751$. So the OLS coefficient of 793.6 sits within 135 of the ATT and roughly7,500 away from the ATE — and on the *opposite side of zero*. Reading 793.6 as"the effect of training on earnings" would get the sign wrong for the population.The `delta` of $-0.971$ that the package reports is the gap between the OLSweight on ATT and $\rho$; it is close to $-1$, its extreme value.Słoczyński's paper itself is an algebraic decomposition of the OLS coefficient $\tau$ in terms of ATT/ATU weights; it does not propose or endorse the propensity-score-as-covariate estimator below as a general-purpose way to recover ATT/ATU/ATE. What follows is our own illustrative exercise, done in three steps, to see how such an estimator would behave in practice: First, estimate the propensity score equation, basically$$ d = X \gamma $$ {#eq-ols-ate-4}Second, include the predicted propensity score as a covariate and estimate the model with treatment and control group separately. Note that regressing $Y$ linearly on the scalar propensity score $p(X)$ assumes $E[Y(d) \mid p(X)]$ is linear in $p(X)$ — a strong and generally untested assumption, unlike matching/weighting directly on the propensity score, or regression adjustment on the covariates $X$ themselves, which are the standard approaches. Treat the numbers below as an exploratory illustration rather than a recommended estimator.Third, predict the counterfactuals for the full sample. Then we get the ATE, ATT and ATU.One more reason to treat this as illustration rather than method: the propensity score here comes from a linear probability model, and on these data it runs from $-0.034$ to $0.154$ --- some fitted "probabilities" are negative. That is harmless when the score is used merely as a regressor, as it is here, but it would be fatal if the same score were used as an inverse-probability weight.Let's see with the same data:```{r}#| label: chunk2m_propensity <-lm(treated ~ age + age2 + educ + black + hispanic + married + nodegree + re74 + re75, data=nswcps)ps <-predict(m_propensity)df_combined <-data.frame(nswcps, ps)df.1<- df_combined[which(treated ==1),]df.0<- df_combined[which(treated ==0),]m_ot <-lm(re78 ~ ps, data = df.1)ot <-suppressWarnings(predict(m_ot, newdata = df_combined)) # add newdata option to predict for full samplem_oc <-lm(re78 ~ ps, data = df.0)oc <-suppressWarnings(predict(m_oc, newdata = df_combined)) # add newdata option to predict for full samplete <- ot - ocate <-mean(te)df_combined$te <- teatt <-as.numeric(mean(df_combined[which(treated ==1),'te']))atu <-as.numeric(mean(df_combined[which(treated ==0),'te']))print(paste("ate=", signif(ate, 4)))print(paste("att=", signif(att, 4)))print(paste("atu=", signif(atu, 4)))```This reproduces the package's decomposition exactly: ATE $-6751$, ATT 928.4, ATU$-6840$, the same three numbers to four significant figures.We see in this case OLS coefficient is hugely different from ATE. This is because $P(d=1)$ is only .01, about 1 percent; we have much larger control group than treatment group. In this case, the OLS coefficient is far different from ATE; in fact, it's pretty close to ATT.Now, what's the intuition that OLS coefficient is just the opposite as the ATE in terms of weighting ATT and ATU? The reason is that OLS is to predict actual outcome, not counterfactual outcome. For calculating ATE, we need counterfactual outcome predicted. Now suppose we have a lot of control observations, we'd much better predicting $(y(0)| d==0)$ rather than $(y(1) | d==1)$. That is, we are better getting the slope of $(y(0) | d==0, X=x)$; therefore predicting $(y(0) | d==1, X=x)$. Therefore although treatment group has less observations, but because the $(y(0) | d==1)$ is better predicted, the ATT is better estimated, thus more weight. If the opposite happens, then the ATU should be more heavily weighted. Thus the counter-intuitive weighting.## Regression adjustmentSłoczyński's method is to include propensity score as a covariate. This propensity score serves as a proxy as all covariates. As Rosenbaum and Rubin (1983) showed, the propensity score is a balancing score (treatment is independent of the covariates given the propensity score). Another way to allow treatment effect to differ across all covariates is to allow interaction of treatment dummy with all covariates, then I am doing "Regression Adjustment" (see Stata's `teffects ra`, used in the [treatment effects and matching chapter](treatment-matching.qmd); note this is not `treatreg`, which was the old endogenous-treatment command, superseded by `etregress`). We can calculate ATE, ATT, ATU after the linear model.```{r}#| label: chunk3library(tidyverse)lm1 <-lm(re78 ~ treated*(age + age2 + educ + black + hispanic + married + nodegree + re74 + re75), data = nswcps)y1 <-predict(lm1,newdata=nswcps %>%mutate(treated=1))y0 <-predict(lm1,newdata=nswcps %>%mutate(treated=0))ate=mean(y1-y0) #ateprint(paste("ate=", signif(ate, 4)))y1.att <-predict(lm1,newdata=nswcps %>%filter(treated==1))y0.att <-predict(lm1,newdata=nswcps %>%filter(treated==1) %>%mutate(treated=0))att=mean(y1.att)-mean(y0.att) #attprint(paste('att=', signif(att, 4)))y1.atu <-predict(lm1,newdata=nswcps %>%filter(treated==0) %>%mutate(treated=1))y0.atu <-predict(lm1,newdata=nswcps %>%filter(treated==0))atu=mean(y1.atu)-mean(y0.atu) #atuprint(paste('atu=', signif(atu, 4)))```Regression adjustment gives ATE $-4930$, ATT 796 and ATU $-4996$.We see this is somewhat different from Słoczyński's method, which gave $-6751$,928.4 and $-6840$: the two agree on sign and order of magnitude but not onlevel, unsurprisingly, since one regresses on a scalar propensity score and theother interacts the treatment with all nine covariates. But the point is the original OLS coefficient 794 is nowhere close to ATE under either approach — $-4930$ or $-6751$, both far away and both negative. In this case, it's very close to ATT (796 by regression adjustment, 928 by the propensity-score route), as the treatment group is very small.---<!-- see-also-footer -->*Systematic treatment: [R](https://xiangao.github.io/causal_econometrics_guide/estimation.html) · [Julia](https://xiangao.github.io/causal_econometrics_julia/estimation.html).*