# Endogeneity
```{r}
#| include: false
library(MASS)
library(ivreg)
```
When we discuss the linear model we assume that the regressors are
exogenous. That word covers conditions of different strength -- full
independence, mean independence, and mere zero correlation -- which the
[OLS chapter](ols.qmd) sets out as a ladder. What instrumental variables
work with is the weakest of them, the moment condition $E[Zu]=0$. However, there could be reasons to believe that some
regressors are correlated with the error term. In that case we call
those regressors endogenous.
Under the classical assumptions OLS estimators are unbiased and
consistent. One key assumption is that the regressors have to be
uncorrelated with the error term. If this condition does not hold,
OLS estimators are biased and inconsistent.
When one independent variable does not satisfy this condition, we say
this variable is endogenous. The estimates of other coefficients are then
usually biased as well, but not always: the contamination travels through the
correlation between regressors, so a coefficient on a regressor orthogonal to
the endogenous one is left alone. With an orthogonal design the damage stays
in the endogenous coefficient.
The most popular cure for endogeneity is to use instrumental variables.
## Instrumental Variables (IV)
The basic idea of IV is to use an exogenous variable (or exogenous variables) which is correlated with the endogenous independent variable as an "instrument" for the endogenous variable.
Suppose the linear regression model
$$
{\symbf{y=X\beta+u, \quad {E} (u u')=\sigma^2 I,
}}$$ {#eq-endogeneity-1}
at least one of the explanatory variables in the $n \times k$
matrix ${\symbf{X}}$ is assumed not to be predetermined with respect to the
error terms, or say, endogenous.
Suppose we have a set of variables ${\symbf{Z}}$, an $n \times l$
matrix of instruments , which satisfies the moment
condition
$$
{{\mathrm{E}}} [{\symbf{Z'(y-X\beta)]=0.
}}$$ {#eq-endogeneity-2}
That is, ${\symbf{Z}}$ is uncorrelated with the error term.
Here is what we do for two-stage least squares (2sls):
Stage 1: Regress each of the variables in the ${\symbf{X}}$ matrix on
${\symbf{Z}}$ to obtain a matrix of fitted values ${\symbf{\hat X}}$,
$$
{\symbf{\hat X=Z(Z'Z)^{-1}Z'X=P_ZX
}}$$ {#eq-endogeneity-3}
This is essentially to get the part of ${\symbf{X}}$ that is correlated with ${\symbf{Z}}$. Or to say, to project ${\symbf{X}}$ on to ${\symbf{Z}}$.
Stage 2: Regress ${\symbf{y}}$ on ${\symbf{\hat X}}$ to obtain the estimated
${\symbf{\beta}}$
$$
{\symbf{ \hat \beta_{2sls}=(\hat X' \hat X)^{-1}(\hat X'
y)=(X'P_ZX)^{-1}(X'P_Zy)=\hat \beta_{IV}
}}$$ {#eq-endogeneity-4}
Standard errors:
$$
Var {\symbf{[ \hat \beta_{2sls}]= \hat \sigma^2 (\hat X' \hat X)^{-1}= \hat
\sigma^2 (X'P_ZX)^{-1}
}}$$ {#eq-endogeneity-5}
where $\hat \sigma^2 ={{\symbf{\hat u' \hat u}}} / N$. This is the
homoskedastic large-sample variance; finite-sample formulas typically
apply a degrees-of-freedom correction ($N-k$ in place of $N$), and
when errors are heteroskedastic or clustered the robust/clustered IV
covariance matrix replaces $\hat\sigma^2 (X'P_ZX)^{-1}$ with a
sandwich form.
Note that ${\symbf{\hat u = y- X \hat \beta_{2sls}}}$.
If we do it manually by two steps, the second step will report a wrong
standard error. That is because the second stage regression will
report standard errors based on ${\symbf{\hat u = y- \hat X \hat
\beta_{2sls}}}$. Therefore, it's always recommended to ask the
statistical program to do a 2sls for you, since presumably that will
give you correct standard errors.
Geometry Illustration:
{width=70%}
Suppose the simplest case: $y=\beta_0 + \beta_1 x + u$. Take all variables in
deviations from their means, so the intercept drops out and $z'\iota = 0$; this
keeps the algebra below about the slope alone. Split $x$ into the part
that lies along the error and the part that does not,
$$ x = x^{\perp} + x^{\parallel}, $$ {#eq-endogeneity-6}
where $x^{\parallel}$ is the orthogonal projection of $x$ onto $u$ and
$x^{\perp}$ is what is left over. Two facts hold *by construction* and are what
make the picture work: $x^{\perp}$ is perpendicular to $u$, and $x^{\perp}$ is
perpendicular to $x^{\parallel}$. Endogeneity is simply the statement that
$x^{\parallel} \neq 0$. (This is a decomposition of $x$, not an extra
assumption about it -- every $x$ admits it.)
Now let $z$ be a vector perpendicular to $u$ with a non-zero projection onto $x$
(relevance: $z$ actually predicts the regressor). Project both $y$ and $x$ onto
$z$, and write $\hat y_z$ and $\hat x_z$ for those two projections. The
instrumental variables estimator is the ratio of their signed projection
coefficients -- signed, because lengths are never negative and $\beta_1$ may be,
$$ \hat \beta_{IV} = \frac{\hat y_z}{\hat x_z} = \frac{z'y}{z'x}, $$ {#eq-endogeneity-7}
since $z'y = \beta_1 z'x + z'u$. The figure draws $z$ exactly perpendicular to
$u$, which sets $z'u = 0$ and removes the second term. Read that as the
idealisation it is. Exogeneity is a statement about the population,
$E[z_i u_i] = 0$, and it does not make $z'u$ vanish in a realised sample. What
we have in a sample of size $n$ is
$$ \hat \beta_{IV} = \beta_1 + \frac{z'u}{z'x}, $$ {#eq-endogeneity-7a}
so $\hat\beta_{IV}$ is consistent because $n^{-1} z'u \to 0$ while $n^{-1}z'x$
converges to something non-zero -- not because the numerator is identically
zero. The geometry shows why IV works; it does not make IV exact.
Staying inside the idealised picture for a moment: because the two right
triangles involved are similar, $\hat\beta_{IV}$ is the same number you would get
from the *infeasible* regression of $y$ on the exogenous part alone: that regression returns
$$ \frac{x^{\perp\prime}y}{x^{\perp\prime}x^{\perp}} = \beta_1 \frac{x^{\perp\prime}x}{x^{\perp\prime}x^{\perp}} + \frac{x^{\perp\prime}u}{x^{\perp\prime}x^{\perp}} = \beta_1, $$ {#eq-endogeneity-8}
where the second term vanishes because $x^{\perp} \perp u$ and the first
collapses to $\beta_1$ because $x^{\perp} \perp x^{\parallel}$, so
$x^{\perp\prime}x = x^{\perp\prime}x^{\perp}$. That infeasible regression is the
one we would run if we could observe the decomposition. We cannot -- which is
exactly why we need the instrument.
```{r}
## DGP: data$y <- data$x + data$z + data$u
library(MASS)
set.seed(66)
nobs=10000
nDim = 3
sdxx = 1
sdww=1
sdzz=1
## here we have three variables x,z,w.
## z is the omitted variable,x and z are correlated, w is the instrument, which is correlated with x, but not z. u is independent of everything else.
crxz=.6
crzw=0
crxw=.5
covarMat = matrix( c(sdxx^2, crxz, crxw, crxz, sdzz^2, crzw, crxw, crzw, sdww^2 ) , nrow=nDim , ncol=nDim )
covarMat
data = data.frame(mvrnorm(n=nobs, mu=rep(0,nDim), Sigma=covarMat ))
names(data) <- c('x','z','w')
data$u <- rnorm(nobs,0,1)
# dgp
data$y <- data$x + data$z + data$u
lm <- lm(y~x, data=data)
lm.full <- lm(y~ x + z, data=data)
tsls.model <- ivreg(y ~ x | w, data=data)
# lm is biased
summary(lm)
# lm.full is good
summary(lm.full)
# tsls is good.
summary(tsls.model)
```
## Control Function Approach
A second way to do an IV regression is also two-step approach: Regress
${\symbf{X}}$ (endogenous) on ${\symbf{Z}}$, get the residual: ${\symbf{\hat
v=X-Z(Z'Z)^{-1}Z'X}}$, then regress $y$ on ${\symbf{X}}$ and $\hat v$ to get
$\hat \beta_{IV}$.
The difference between this approach and the 2sls approach is that in
2sls we regress $y$ on ${\symbf{\hat X}}$; in the control function approach,
we regress $y$ on ${\symbf{ X}}$ and $\hat v$. They should both give you
the same coefficient estimates. There are advantages of using the
control function approach.
An example to show how to do 2sls manually, or using control function approach.
```{r}
## DGP: data$y <- data$x + data$z + data$u
lm1 <- lm(y~x, data=data)
lm2 <- lm(x~w, data=data)
tsls.manual <- lm(data$y ~ lm2$fitted.values)
summary(tsls.manual)
# control function approach
tsls.control <- lm(data$y ~ data$x + lm2$residuals)
summary(tsls.control)
```
This approach can be used to do a simple endogeneity test. First, you
can do a simple test of endogeneity of ${\symbf{X}}$. For example:
reg x_endog x* z*
predict v_x, resid
reg y x_endog x* v_x
test v_x
Obviously, this test is based on ${\symbf{Z}}$ being exogenous.
Residual inclusion extends to *some* non-linear models -- logit,
probit, Poisson -- under additional control-function assumptions, but
it is not automatically valid for "any glm". The construction is
model-specific: the appropriate first-stage residual (a plain linear
residual or a generalized residual), the identification conditions,
and the standard errors all differ for continuous, binary, and
nonlinear endogenous regressors, and inserting a linear first-stage
residual into an arbitrary GLM is generally not correct. The
generated-regressor step also inflates uncertainty, so standard errors
need to account for the estimated first stage (e.g. by bootstrap).
For example, the "eteffects" procedure in Stata (version 14) uses a
control-function approach to get endogenous treatment effects for
several outcome types.
## Durbin-Wu-Hausman Test
### Idea
In econometric modeling, there are often questions on endogeneity. Do
we know how to test whether an independent variable is endogenous
statistically? The answer is: sort of, but not really. We cannot do
endogeneity test without a valid instrument. Therefore, we have to
have strong argument for a valid instrument first before we can do
endogeneity test.
With endogenous variables on the right-hand side of the equation, we
need to use instrumental variable (IV) regression for consistent
estimation. However, with IV regression, we lose efficiency: the
asymptotic variance of the IV estimator is larger, and can be much
larger than the OLS estimator. Therefore, we gain consistency, but
lose efficiency, by using IV estimator when there is an endogeneity
problem.
Now we have a familiar scenario (if you are familiar with Hausman
test for fixed effect and random effect estimator for panel data):
Suppose we have the null hypothesis as the regressor being exogenous.
We have an efficient estimator under null hypothesis yet inconsistent
under alternative hypothesis (OLS estimator). We also have a
consistent estimator under both null and alternative (IV estimator).
Similar to panel data setting, we have the Hausman test statistic as:
$$ H = (\hat \beta_c - \hat \beta_e)' D^{-} (\hat \beta_c - \hat
\beta_e) $$ {#eq-endogeneity-9}
where $D={{\mathrm{Var}}} [\hat \beta_c] - {{\mathrm{Var}}} [\hat \beta_e]$, $^-$ is
the generalized inverse, $\hat \beta_c$ is the consistent estimator (in
this case the IV estimator) and $\hat \beta_e$ is the efficient
estimator (in this case OLS estimator).
$H$ conforms to $\chi^2_k$ asymptotically, where $k$ is the number of endogenous
variables.
This test is to compare the IV estimator and the OLS estimator: if
it's close, then OLS estimator is fine (fail to reject null that OLS
is consistent, or say the variable is exogenous). If it's large, then
IV estimator is needed, although we lose some efficiency. This test
is based on the assumption that the instruments are exogenous.
If that is in question, then it's pointless to do the test, since the
IV estimator cannot guarantee consistency either.
### Implementation in Stata
In Stata, there are different ways to do it:
Do a regular Hausman test:
ivreg y x1 (x2=x3 x4)
estimates store iv
reg y x1 x2
hausman iv ., constant sigmamore
Or, simply use "ivendog" in Stata.
## Identification
Identification in a regression equation means that all parameters can
be uniquely estimated. The relevant counting rule (the *order
condition*) is most cleanly stated in terms of *excluded* instruments.
Let $k_e$ be the number of endogenous regressors and $l_e$ the number
of excluded instruments (instruments not already included as
exogenous regressors). A necessary condition is $l_e \ge k_e$:
there must be at least as many excluded instruments as endogenous
regressors. (Equivalently, counting the included exogenous regressors
as their own instruments, the total number of instruments must be at
least the total number of regressors.) If $l_e=k_e$ we have
exact-identification. If $l_e>k_e$, we have over-identification.
Over-identification generates more efficient estimates, given the
assumption of instruments being exogenous. The other advantage of
over-identification is that over-identification tests can be done to
test the adequacy of instruments.
Under the null hypothesis that all the instruments are uncorrelated
with the error term, an LM statistic $N \times R^2$ conforms to
$\chi^2 (r)$ distribution, $r=l_e-k_e$, the number of excess instruments,
or say, the number of overidentifying restrictions. If we reject the null,
then we should be concerned about the exogeneity of the whole set of
the instruments. This test is called Sargan's test in IV context, and
(Hansen's) J test in GMM context. The two names are not quite
interchangeable: Sargan's $N \times R^2$ form assumes homoskedastic errors,
while Hansen's $J$, built from the efficient GMM weighting, is the
heteroskedasticity-robust counterpart and the one to use whenever the standard
errors are robust.
It is worth being exact about what the test delivers, because it is easy to
over-read. It is a joint test of the overidentifying restrictions. Failing to
reject does not certify the instruments: the test has no power against
violations that all of them share, because instruments wrong in the same way
agree on the same wrong answer. Rejection, for its part, does not say *which*
moment condition failed. A passing $J$ is the absence of one particular kind
of evidence against the instruments, not evidence for them.
What the J test or Sargan's test does is to test the whole set of
instruments being exogenous or not. There is another test for
testing exogeneity for a subset of instruments. It's call a C test
or a difference-in-Sargan test. The idea is to calculate the
difference between two Sargan's statistics (or Hansen's J in GMM
setting); one is with the whole set of instruments, the other one
without the suspected instruments. The null is that the suspect
instruments are exogenous; or orthogonal to the error term.
Obviously to conduct the C test, we'll have to have at least one
extra instrument more than the number of endogenous variables.
To understand it better, we look at how to implement Sargan's test
manually: For the 2SLS estimator, the test statistic is Sargan's
statistic, typically calculated as $N \times R^2$ from a regression of
the IV residuals on the full set of instruments.
. ivregress 2sls rent pcturban (hsngval = faminc i.region)
. estat overid
Tests of overidentifying restrictions:
Sargan (score) chi2(3) = 11.2877 (p = 0.0103)
Basmann chi2(3) = 12.8294 (p = 0.0050)
. predict res, residual
. reg res pcturban faminc i.region
. disp e(N)*e(r2)
11.287665
### Implementation in Stata
In Stata, there are different ways to do over-identification test,
ivreg2 reports a comprehensive set of tests; overid command does the
over-identification test after the ivreg command.
ivreg2 with gmm option returns J test; it reports
Sargan's test without this option.
ivreg2 also reports the C test statistic, with ortho(). If the C test rejects and the J test without the suspect instruments does not, the evidence points at the suspect instruments -- conditional on the retained ones being valid. That condition does real work: a difference test cannot tell an invalid suspect from an invalid maintained instrument, so it locates the problem only when the maintained set is sound.
## Weak Instruments
### Problem with the cure
An instrument needs to satisfy two criteria: orthogonality and
relevance. We need instruments to be orthogonal to the error term.
We can verify the orthogonality condition by Sargan's test if there are extra
instruments.
It turns out instrument relevance is important too: if instruments are
weak, the usual large-sample approximations are poor. It helps to
keep three cases apart. With fixed, nonzero first-stage relevance, IV
remains *consistent* but can be badly *biased* in finite samples (and
biased toward OLS), with the normal approximation working poorly.
Under weak-instrument (local-to-zero) asymptotics, the limiting
distribution is *nonstandard* and not centered in the usual way, so
conventional standard errors and $t$-statistics mislead. Only in the
limiting case of complete irrelevance ($\Pi=0$) is the parameter
unidentified.
To see the problem, suppose
$$
{\symbf{y=X\beta+u, \quad {E} (u u')=\sigma_u^2 I,
}}$$ {#eq-endogeneity-10}
$$
{\symbf{X=Z\Pi+v, \quad {E} (v v')=\sigma_v^2 I,
}}$$ {#eq-endogeneity-11}
and
$$
{{\mathrm{E}}}({\symbf{Z u )=0.
}}$$ {#eq-endogeneity-12}
We can see here ${\symbf{Z}}$ is exogenous. However, the model does not say
anything about relevance. To illustrate the problem caused by weak
instrument, suppose we have only one endogenous variable and one
instrument.
$$
\hat \beta_{2sls}=\frac{{\symbf{Z' y}}}{{\symbf{Z' X}}} = \frac{{\symbf{Z'(X \beta +
u)}}}{{\symbf{Z'
X}}}= \beta + \frac{{\symbf{Z' u}}}{{\symbf{Z' X}}} .
$$ {#eq-endogeneity-13}
If ${\symbf{Z}}$ is irrelevant, or, $\Pi=0$, then
$$
\hat \beta_{2sls}- \beta=\frac{{\symbf{Z' u}}}{{\symbf{Z' v}}} =\frac{ \frac{1}{\sqrt n}
\sum_{i=1}^n Z_i u_i}{\frac{1}{\sqrt n} \sum_{i=1}^n Z_i v_i}
\xrightarrow{d} \frac{z_u}{z_v} ,
$$ {#eq-endogeneity-14}
where (normalizing ${{\mathrm{Var}}}(Z_i)=1$, which is harmless since any
common scale factor cancels in the ratio)
$$
\begin{bmatrix}
z_u \\z_v
\end{bmatrix}
\sim N(0, \begin{bmatrix}
\sigma_u^2 \quad \sigma_{uv} \\ \sigma_{uv} \quad \sigma_v^2
\end{bmatrix}).
$$ {#eq-endogeneity-15}
Therefore, if ${\symbf{Z}}$ is irrelevant, $\beta_{2sls}$ is inconsistent.
Also, the distribution of the bias is Cauchy-like (the ratio of
correlated normals).
This is a case where the cure might be worse than the disease itself:
the bias can be big comparing to the bias an OLS estimate suffers.
### Weak Instrument Tests
There are a variety of weak-instruments tests proposed. Most of them
are based on so-called weak-instruments asymptotics and a new
parameter called "concentration parameter" $\mu^2 = \Pi' Z' Z \Pi
/ \sigma_v^2$. Sample size only enters the distribution through
$\mu^2$.
With weak-instruments asymptotics, IV estimators are no longer
consistent, and they are not normal asymptotically. Most test
statistics (J test, etc.) do not have normal or $\chi^2$ distributions
anymore.
Now I list the following tests in the order of recommended level by
James Stock:
1. Moreira (2003) conditional likelihood ratio test (CLR).
Advantages of this test:
a. Nearly optimal among invariant similar tests (Andrews, Moreira and Stock 2006).
b. Implemented in Stata as condivreg.
Disadvantages:
a. Complicated. Only developed so far for one endogenous
variable case.
2. Stock-Yogo bias method and size method.
Stock and Yogo (2005) provide critical values for both methods: one is
to control the size of bias, the other one is to control the size of a
Wald test of $\beta=\beta_0$. Bias method is more frequently used.
In the case of multiple endogenous variables, the Cragg-Donald
statistics is used to compare with the critical values. It is
implemented in Stata as part of the *ivreg2* command, but it's
only available for the situation there are at least two excluded
variables (meaning the number of instruments minus the number of
endogenous variables).
3. Anderson-Rubin confidence intervals.
In the model of
$$
{\symbf{y=X\beta+u, \quad {E} (u u')=\sigma_u^2 I,
}}$$ {#eq-endogeneity-16}
$$
{\symbf{X=Z\Pi+v, \quad {E} (v v')=\sigma_v^2 I,
}}$$ {#eq-endogeneity-17}
The null hypothesis $H_0: \beta=\beta_0$. Anderson-Rubin statistic
is the F statistic in the regression of $y-X \beta_0$ on $Z$, the F
test that the coefficient vector on $Z$ is zero (under the model that
coefficient equals $\Pi (\beta - \beta_0)$, so it is zero exactly when
$\beta=\beta_0$):
$$
AR(\beta_0)= \frac{(y-X\beta_0)' P_Z (y-X\beta_0)/l}{(y-X\beta_0)' M_Z
(y-X\beta_0)/(N-l)}
$$ {#eq-endogeneity-18}
where $l$ is the number of instruments. The idea of AR confidence
interval is to collect all values of $\beta_0$ at which we fail to
reject $H_0: \beta=\beta_0$.
4. First-stage F test.
The familiar rule of thumb is $F>10$, and its scope is worth knowing. It comes
from the single-endogenous-regressor case with homoskedastic errors, where it
corresponds roughly to a bound on the relative bias of 2SLS. The formal
Stock-Yogo critical values depend on how many instruments there are and on which
criterion -- relative bias or size distortion -- is being controlled, so one
number does not carry over to every design. Under heteroskedasticity or
clustering the relevant statistic is the effective first-stage F of Montiel Olea
and Pflueger rather than the conventional one. And when identification is weak
the dependable route is not a higher threshold but weak-instrument-robust
inference -- the Anderson-Rubin and CLR tests in this list, whose size does not
depend on instrument strength.
5. Kleibergen's LM test.
This test is dominated by the CLR test, thus no longer the optimal
test to use.
6. First-stage $R^2$, or partial $R^2$, etc., are not recommended.
## When is endogeneity a problem and what can we say from an IV regression
Let's consider a model of housing price on no. of rooms
and square footage.
$$ P = \alpha R + \beta F + \epsilon $$ {#eq-endogeneity-19}
Should square footage really be there? We are not interested in,
given square footage, how much money we need to pay for an additional
room. We are generally interested in how much money we need to pay
for an additional footage. However, if we do not include footage,
then we "sort of" have endogeneity problem due to omitted variable.
Another example would be education's impact on wage. Say IQ is an
omitted variable. Do we want to say something about education's
impact on wage controlling on IQ, or not?
So it depends on what questions to ask. If I am a builder and ask
what an extra bedroom on the already built house would bring to me, I
need to control for footage. If I am generally asking what an extra
bedroom would cost, then I am implying what that bedroom and
associated footage would cost me. In that case, without controlling
for footage is not a problem, since the question is: what an
additional bedroom (AND the associated average square footage) would
cost me.
In the wage and education example, if I am asking: what's the effect
of education on wage? I implicitly ask what would be my pay raise if
I go to college (given IQ, of course), for example. Then we are
concerned that this model of
$$ wage= \alpha edu + \epsilon $$ {#eq-endogeneity-20}
will generate biased estimate of $\alpha$, because we are not
controlling for IQ. But if I am an employer and interested in hiring
a person, what would I have to pay extra to hire a college graduate
vs. a high school graduate? Then I don't have to include that IQ
variable, because implicitly I am asking: what extra I need to pay for
that college graduate who comes with a higher IQ, and everything else
that is associated with higher education.
In addition, what an IV regression really gives us is part of the causal
inference of $X$ (suspected to be endogenous) on $y$, namely the part
induced by $Z$. For example, we are interested in the relationship
between college education and wage. If we use distance to college as
an instrument, then our inference is the effect of college education
from those who decide to go because of proximity of the college.
This is the Local Average Treatment Effect (LATE) of Imbens and Angrist
(1994): with a binary instrument and heterogeneous treatment effects,
IV identifies the causal effect for *compliers* --- those whose
treatment status is shifted by the instrument --- rather than for the
full population. The formal derivation, including the monotonicity
assumption and the complier/always-taker/never-taker decomposition, is
in the [Causal Econometrics Guide](https://xiangao.github.io/causal_econometrics_guide/iv-rdd.html).
