Bias is large only when the omitted variable is strongly associated with both \(Y\) and \(D\). The Cinelli and Hazlett framework yields three key quantities.
15.2 Partial \(R^2\) of treatment on outcome
This measures how much variation in the outcome the treatment explains after partialling out \(X\). In the extreme case where the confounder explains all remaining outcome variance (\(R^2_{Y \sim Z|D,X} = 1\)), the treatment effect goes to zero only if \(R^2_{D \sim Z|X} \ge R^2_{Y \sim D|X}\). In other words, the confounder’s association with treatment must exceed the treatment’s partial association with the outcome.
15.3 Robust value
\(RV_{q,\alpha}\) is the minimum strength of association the confounder must have with both treatment and outcome for a \((1-\alpha)\) confidence interval to include a \(q\%\) change from the current estimate. With \(q=1\) this gives the threshold for the effect to include zero. Both the \(Z\)-\(D\) and \(Z\)-\(Y\) associations must exceed this threshold simultaneously.
15.4 Bounds using observed covariate(s)
The first two quantities are dimensionless but hard to interpret without context. Benchmarking against an observed covariate helps. If a covariate \(X_j\) is known to be an important predictor, we can ask how much stronger the confounder must be relative to \(X_j\) to drive the effect to zero.
Sensitivity Analysis to Unobserved Confounding
Model Formula: peacefactor ~ directlyharmed + age + farmer_dar + herder_dar +
pastvoted + hhsize_darfur + female + village
Null hypothesis: q = 1 and reduce = TRUE
-- This means we are considering biases that reduce the absolute value of the current estimate.
-- The null hypothesis deemed problematic is H0:tau = 0
Unadjusted Estimates of 'directlyharmed':
Coef. estimate: 0.0973
Standard Error: 0.0233
t-value (H0:tau = 0): 4.1844
Sensitivity Statistics:
Partial R2 of treatment with outcome: 0.0219
Robustness Value, q = 1: 0.1388
Robustness Value, q = 1, alpha = 0.05: 0.0763
Verbal interpretation of sensitivity statistics:
-- Partial R2 of the treatment with the outcome: an extreme confounder (orthogonal to the covariates) that explains 100% of the residual variance of the outcome, would need to explain at least 2.19% of the residual variance of the treatment to fully account for the observed estimated effect.
-- Robustness Value, q = 1: unobserved confounders (orthogonal to the covariates) that explain more than 13.88% of the residual variance of both the treatment and the outcome are strong enough to bring the point estimate to 0 (a bias of 100% of the original estimate). Conversely, unobserved confounders that do not explain more than 13.88% of the residual variance of both the treatment and the outcome are not strong enough to bring the point estimate to 0.
-- Robustness Value, q = 1, alpha = 0.05: unobserved confounders (orthogonal to the covariates) that explain more than 7.63% of the residual variance of both the treatment and the outcome are strong enough to bring the estimate to a range where it is no longer 'statistically different' from 0 (a bias of 100% of the original estimate), at the significance level of alpha = 0.05. Conversely, unobserved confounders that do not explain more than 7.63% of the residual variance of both the treatment and the outcome are not strong enough to bring the estimate to a range where it is no longer 'statistically different' from 0, at the significance level of alpha = 0.05.
Bounds on omitted variable bias:
--The table below shows the maximum strength of unobserved confounders with association with the treatment and the outcome bounded by a multiple of the observed explanatory power of the chosen benchmark covariate(s).
Bound Label R2dz.x R2yz.dx Treatment Adjusted Estimate Adjusted Se
1x female 0.0092 0.1246 directlyharmed 0.0752 0.0219
2x female 0.0183 0.2493 directlyharmed 0.0529 0.0204
3x female 0.0275 0.3741 directlyharmed 0.0304 0.0187
Adjusted T Adjusted Lower CI Adjusted Upper CI
3.4389 0.0323 0.1182
2.6002 0.0130 0.0929
1.6281 -0.0063 0.0670
\(RV_{q=1} = 13.9 \%\). This means that unobserved confounders that explain 13.9% of the residual variance both of the treatment and of the outcome are sufficiently strong to explain away all the observed effect. On the other hand, unobserved confounders that do not explain at least 13.9% of the residual variance both of the treatment and of the outcome are not sufficiently strong to do so.
The robustness value for testing the null hypothesis that the coefficient of “directlyharmed” is zero (\(RV_{q=1,\alpha=0.05}\)) is 7.6%. This means that unobserved confounders that explain 7.6% of the residual variance both of the treatment and of the outcome are sufficiently strong to bring the lower bound of the confidence interval to zero (at the chosen significance level of 5%).
The partial \(R^2\) of directlyharmed with peacefactor is 2.2%. This means that, in an extreme scenario in which we assume that unobserved confounders explain all of the left-out variance of the outcome, these unobserved confounders would need to explain at least 2.2% of the residual variance of the treatment to fully explain away the observed effect.
The above numbers are for the absolute strength the confounder needs to be to explain away the treatment effect. The authors believe in this context, “female” is an important factor. Suppose the confounder is as important as “female”, what the partial \(R^2\) would be? It is shown that \(R^2_{Y \sim Z|X,D} = 12.5 \%\) and \(R^2_{D \sim Z|X} = 0.9 \%\), so a confounder as strong as “female” is not strong enough to explain away the treatment effect.
It is worth being careful about why, because comparing each number to \(RV_{q=1} = 13.9\%\) one at a time invites the wrong reading. The \(RV\) is defined for a confounder of equal strength on both sides, so it is not a threshold that each partial \(R^2\) must separately clear. Here the outcome-side association (12.5%) is in fact close to the \(RV\), which on its own might look like a near miss — but the treatment-side association is only 0.9%, more than an order of magnitude short. It is the treatment side that does the work: “female” predicts the outcome nearly well enough to matter, and barely predicts the treatment at all. The contour plot below is the honest check, since it evaluates the pair jointly rather than either coordinate alone.
Code
# plot bias contour of point estimateplot(sensitivity)
This plot shows that even 3 times as strong as “female”, the confounder is not strong enough to explain away the treatment effect.
15.6 sensitivity analysis for IV
Even with an IV model, we need sensitivity analysis. The worry would be there could be an unobserved confounder that affects both the instrument and the outcome.
Cinelli and Hazlett have a package “iv.sensemakr” for this. The usage is similar to “sensemakr”.
The example is Card’s returns-to-schooling data, 3,010 men from the 1976 NLS. The outcome is log wage, the treatment is years of education, and the instrument is nearc4, an indicator for growing up near a four-year college. The covariates are experience and its square, race, region, and urban indicators. The concern the sensitivity analysis addresses is a confounder affecting both proximity to a college and wages.
Code
# loads packagelibrary(iv.sensemakr)# loads datasetdata("card")# prepares datay<-card$lwage# outcomed<-card$educ# treatmentz<-card$nearc4# instrumentx<-model.matrix(~exper+expersq+black+south+smsa+reg661+reg662+reg663+reg664+reg665+reg666+reg667+reg668+smsa66, data =card)# covariates# fits IV modelcard.fit<-iv_fit(y,d,z,x)# see resultscard.fit
Instrumental Variable Estimation
(Anderson-Rubin Approach)
=============================================
IV Estimates:
Coef. Estimate: 0.132
t-value: 2.33
p-value: 0.02
Conf. Interval: [0.025, 0.285]
Note: H0 = 0, alpha = 0.05, df = 2994.
=============================================
See summary for first stage and reduced form.
Code
# runs sensitivity analysiscard.sens<-sensemakr(card.fit, benchmark_covariates =c("black", "smsa"))# see resultscard.sens
Sensitivity Analysis for Instrumental Variables
(Anderson-Rubin Approach)
=============================================================
IV Estimates:
Coef. Estimate: 0.132
t-value: 2.33
p-value: 0.02
Conf. Interval: [0.025, 0.285]
Sensitivity Statistics:
Extreme Robustness Value: 0.000523
Robustness Value: 0.00667
Bounds on Omitted Variable Bias:
Bound Label R2zw.x R2y0w.zx Lower CI Upper CI Crit. Thr.
1x black 0.00221 0.0750 -0.0212 0.402 2.59
1x smsa 0.00639 0.0202 -0.0192 0.396 2.57
Note: H0 = 0, q >= 1, alpha = 0.05, df = 2994.
=============================================================
See summary for first stage and reduced form.
The basic idea is to run an IV regression, then apply sensemakr to that model. In this case, the authors use “black” and “smsa” as benchmark covariates. The authors believe these are important factors.
The IV estimate is 0.132 with \(t = 2.33\), \(p = 0.02\), and an Anderson-Rubin confidence interval of \([0.025, 0.285]\) that excludes zero. Read the sensitivity statistics next to that, and the picture is not reassuring. The Robustness Value is 0.00667 — two-thirds of one percent. Compare the OLS example earlier in this chapter, where \(RV_{q=1}\) was 13.9%. This IV result is roughly twenty times more fragile than that one.
The benchmark bounds make it concrete, and they point the other way from what one might hope. A confounder as strong as black explains 0.22% of the residual variance of the instrument and 7.5% of the outcome, and under it the adjusted interval becomes \([-0.021, 0.402]\). For smsa the figures are 0.64% and 2.0%, giving \([-0.019, 0.396]\). Both intervals contain zero. A confounder no stronger than either benchmark covariate is enough to remove the significance of the estimate.
That smsa number is worth a second look on its own terms: its instrument-side partial \(R^2\) of 0.00639 sits just under the Robustness Value of 0.00667, so a confounder of that strength lands essentially exactly on the tipping point. The two diagnostics agree.
None of this says the Card estimate is wrong. It says the identification rests on the instrument being cleaner than black or smsa are as predictors, which is an assumption about college proximity that the data cannot check. We can also plot a contour plot to see how strong the confounder needs to be to explain away the treatment effect.
---title: "Sensitivity analysis of OLS with sensemakr"date: "2025-04-03"---## OVBConsider a true model,$$ Y = \delta D + X \beta + \gamma Z + \epsilon $$ {#eq-sensitivity-analysis-1}If $Z$ is unobserved we estimate$$ Y = \delta D + X \beta + u $$ {#eq-sensitivity-analysis-2}and face omitted variable bias (OVB). Cinelli and Hazlett (2020) show that$$ |\widehat{\text{bias}}| = \sqrt{\frac{R^2_{Y \sim Z|D,X} R^2_{D \sim Z|X}}{1-R^2_{D \sim Z|X}}} (\frac{\hat \sigma_{y.dx}}{\hat \sigma_{d.x}})$$ {#eq-sensitivity-analysis-3}Bias is large only when the omitted variable is strongly associated with both $Y$ and $D$. The Cinelli and Hazlett framework yields three key quantities.## Partial $R^2$ of treatment on outcomeThis measures how much variation in the outcome the treatment explains after partialling out $X$. In the extreme case where the confounder explains all remaining outcome variance ($R^2_{Y \sim Z|D,X} = 1$), the treatment effect goes to zero only if $R^2_{D \sim Z|X} \ge R^2_{Y \sim D|X}$. In other words, the confounder's association with treatment must exceed the treatment's partial association with the outcome.## Robust value$RV_{q,\alpha}$ is the minimum strength of association the confounder must have with both treatment and outcome for a $(1-\alpha)$ confidence interval to include a $q\%$ change from the current estimate. With $q=1$ this gives the threshold for the effect to include zero. Both the $Z$-$D$ and $Z$-$Y$ associations must exceed this threshold simultaneously.## Bounds using observed covariate(s)The first two quantities are dimensionless but hard to interpret without context. Benchmarking against an observed covariate helps. If a covariate $X_j$ is known to be an important predictor, we can ask how much stronger the confounder must be relative to $X_j$ to drive the effect to zero.$$ k_D := \frac{R^2_{D \sim Z|X_{-j}}}{R^2_{D \sim X_j|X_{-j}}} $$ {#eq-sensitivity-analysis-4}where $X_{-j}$ represents all covariates other than the $j$th we pick as the benchmark covariate.$$ k_Y := \frac{R^2_{Y \sim Z|D,X_{-j}}}{R^2_{Y \sim X_j|D,X_{-j}}} $$ {#eq-sensitivity-analysis-5}## ExampleHere is an example from sensemakr documentation.```{r}# loads packagelibrary(sensemakr)# loads datasetdata("darfur")# runs regression modelmodel <-lm(peacefactor ~ directlyharmed + age + farmer_dar + herder_dar + pastvoted + hhsize_darfur + female + village, data = darfur)# runs sensemakr for sensitivity analysissensitivity <-sensemakr(model = model, treatment ="directlyharmed",benchmark_covariates ="female",kd =1:3)# short description of resultssensitivity# long description of resultssummary(sensitivity)``````{r}#| results: 'asis'# minimal reportingovb_minimal_reporting(sensitivity, format ="html")```Let's see how to interpret this. $RV_{q=1} = 13.9 \%$. This means that unobserved confounders that explain 13.9% of the residual variance both of the treatment and of the outcome are sufficiently strong to explain away all the observed effect. On the other hand, unobserved confounders that do not explain at least 13.9% of the residual variance both of the treatment and of the outcome are not sufficiently strong to do so. The robustness value for testing the null hypothesis that the coefficient of "directlyharmed" is zero ($RV_{q=1,\alpha=0.05}$) is 7.6%. This means that unobserved confounders that explain 7.6% of the residual variance both of the treatment and of the outcome are sufficiently strong to bring the lower bound of the confidence interval to zero (at the chosen significance level of 5%).The partial $R^2$ of `directlyharmed` with `peacefactor` is 2.2%. This means that, in an extreme scenario in which we assume that unobserved confounders explain all of the left-out variance of the outcome, these unobserved confounders would need to explain at least 2.2% of the residual variance of the treatment to fully explain away the observed effect.The above numbers are for the absolute strength the confounder needs to be to explain away the treatment effect. The authors believe in this context, "female" is an important factor. Suppose the confounder is as important as "female", what the partial $R^2$ would be? It is shown that $R^2_{Y \sim Z|X,D} = 12.5 \%$ and $R^2_{D \sim Z|X} = 0.9 \%$, so a confounder as strong as "female" is not strong enough to explain away the treatment effect.It is worth being careful about *why*, because comparing each number to $RV_{q=1} = 13.9\%$ one at a time invites the wrong reading. The $RV$ is defined for a confounder of *equal* strength on both sides, so it is not a threshold that each partial $R^2$ must separately clear. Here the outcome-side association (12.5%) is in fact close to the $RV$, which on its own might look like a near miss --- but the treatment-side association is only 0.9%, more than an order of magnitude short. It is the treatment side that does the work: "female" predicts the outcome nearly well enough to matter, and barely predicts the treatment at all. The contour plot below is the honest check, since it evaluates the pair jointly rather than either coordinate alone. ```{r}# plot bias contour of point estimateplot(sensitivity)```This plot shows that even 3 times as strong as "female", the confounder is not strong enough to explain away the treatment effect. ## sensitivity analysis for IVEven with an IV model, we need sensitivity analysis. The worry would be there could be an unobserved confounder that affects both the instrument and the outcome. Cinelli and Hazlett have a package "iv.sensemakr" for this. The usage is similar to "sensemakr".The example is Card's returns-to-schooling data, 3,010 men from the 1976 NLS.The outcome is log wage, the treatment is years of education, and the instrumentis `nearc4`, an indicator for growing up near a four-year college. The covariatesare experience and its square, race, region, and urban indicators. The concernthe sensitivity analysis addresses is a confounder affecting both proximity to acollege and wages.```{r}# loads packagelibrary(iv.sensemakr)# loads datasetdata("card")# prepares datay <- card$lwage # outcomed <- card$educ # treatmentz <- card$nearc4 # instrumentx <-model.matrix( ~ exper + expersq + black + south + smsa + reg661 + reg662 + reg663 + reg664 + reg665+ reg666 + reg667 + reg668 + smsa66,data = card) # covariates# fits IV modelcard.fit <-iv_fit(y,d,z,x)# see resultscard.fit# runs sensitivity analysiscard.sens <-sensemakr(card.fit, benchmark_covariates =c("black", "smsa"))# see resultscard.senssummary(card.sens)```The basic idea is to run an IV regression, then apply sensemakr to that model. In this case, the authors use "black" and "smsa" as benchmark covariates. The authors believe these are important factors.The IV estimate is 0.132 with $t = 2.33$, $p = 0.02$, and an Anderson-Rubinconfidence interval of $[0.025, 0.285]$ that excludes zero. Read the sensitivitystatistics next to that, and the picture is not reassuring. The Robustness Valueis 0.00667 — two-thirds of one percent. Compare the OLS example earlier in thischapter, where $RV_{q=1}$ was 13.9%. This IV result is roughly twenty times morefragile than that one.The benchmark bounds make it concrete, and they point the other way from what onemight hope. A confounder as strong as `black` explains 0.22% of the residualvariance of the instrument and 7.5% of the outcome, and under it the adjustedinterval becomes $[-0.021, 0.402]$. For `smsa` the figures are 0.64% and 2.0%,giving $[-0.019, 0.396]$. **Both intervals contain zero.** A confounder nostronger than either benchmark covariate is enough to remove the significance ofthe estimate.That `smsa` number is worth a second look on its own terms: its instrument-sidepartial $R^2$ of 0.00639 sits just under the Robustness Value of 0.00667, so aconfounder of that strength lands essentially exactly on the tipping point. Thetwo diagnostics agree.None of this says the Card estimate is wrong. It says the identification rests onthe instrument being cleaner than `black` or `smsa` are as predictors, which isan assumption about college proximity that the data cannot check. We can also plot a contour plot to see how strong the confounder needs to be to explain away the treatment effect.```{r}#| results: 'asis'plot(card.sens, parm ="rf", sensitivity.of ="t-value", lim =0.08, kz =1, ky =1)```Unfortunately the package currently only allows for one instrument. Should not be too hard to modify it to be more general.---<!-- see-also-footer -->*Systematic treatment: [R](https://xiangao.github.io/causal_econometrics_guide/sensitivity-analysis.html) · [Julia](https://xiangao.github.io/causal_econometrics_julia/sensitivity-analysis.html).*
