Panel data are repeated observations on the same “unit” (we call that cross-sectional units) over time. Panel data models are used since it has advantages over cross-sectional models or time-series models. The major advantage it has over cross-sectional models is: Controlling for individual heterogeneity.
For example, Baltagi and Levin (1992) studies cigarette demand across 46 states for the years 1963-88. Consumption is modeled as a function of lagged consumption, price and income. These variables vary across states and time. However, there are other variables that may be time-invariant (\(z_i\)) or individual-invariant (\(w_t\)). Examples of \(z_i\) are religion and, in a panel of adults, completed education. For example, Utah, as a Mormon state, has very low cigarette demand due to religious reason. Generally we consider it does not change over time or change very little over time. Examples of \(w_t\) include cigarette commercials on national TV or radio. Panel data models are able to control for individual heterogeneity (or cross-sectional heterogeneity) while cross-sectional models are not. In many (or most) social science studies, there is “unobserved effects” which is embedded in the error term. In other words, we have “omitted variable” problem. Without controlling for it, the estimation results are generally biased and inconsistent.
Panel data give more informative data, more variability and less collinearity among variables. Often time series data suffers from multicollinearity, while panel data has more variability from its cross-sectional units.
The basic unobserved effects model can be written as: \[ y_{it}={{\symbf{{x_{it} \beta}}}} + c_i + u_{it} \tag{9.1}\]
where \(i\) indexes “units” (can be people, firms, or households, etc.), and \(t\) indexes time periods.
Three kinds of regressor appear in this chapter, and it pays to name them once. \({{\symbf{x}}}_{it}\) varies across units and time — price, income, and in most applications the treatment itself. \(z_i\) varies across units only, so within a unit it is fixed over time; religion and completed education above are the examples. \(w_t\) varies across time only, so in any period it is common to every unit; the national TV campaign is one. Which group a variable falls into is what decides almost everything that follows, including which coefficients fixed effects can estimate and how OLS weights the data.
Instruments do not appear until the panel turns dynamic. When they do, they are written \({{\symbf{Z}}}\), which is a different object from the time-invariant \(z_i\) here — see the Endogeneity, GMM and Dynamic Panel Data chapters.
Comparing to regular cross-sectional models, there is an extra term \(c_i\). \(c_i\) can be treated as a random effect or fixed effect. Traditionally it is distinguished by whether it is estimated as a random variable or a parameter. Modern econometrics tends to distinguish by the correlation between \(c_i\) and \({{\symbf{{x_{it}}}}}\). If there is no correlation between \(c_i\) and \({{\symbf{{x_{it}}}}}\), then it’s a random effect; otherwise, it’s a fixed effect.
It is true that in some cases it is hard to justify that there is no correlation between \(c_i\) and \({{\symbf{{x_{it}}}}}\), which is one reason that people in economics tend to use fixed-effect models. However, fixed-effect models have its own limitations. One of them is: It is hard to justify that ALL the cross-sectional units’ characteristics other than those already in the model do not change over time.
A random effect model puts \(c_i\) into the error term, then estimate by FGLS (feasible GLS). Random effect model is sometimes called “Error-components model” since the overall error term is divided into an individual level error term (\(c_i\)) and individual-time level error term (\(u_{it}\)).
Under a random effect model, the variance-covariance matrix of the error term becomes:
\[ {\symbf{\Omega= }}\begin{bmatrix} \sigma_{c}^2 + \sigma_u^2 & \sigma_{c}^2 & \cdots & \sigma_{c}^2 \\ \sigma_{c}^2 & \sigma_{c}^2 + \sigma_u^2 & \cdots & \sigma_{c}^2 \\ \vdots & \vdots & \ddots & \vdots \\ \sigma_{c}^2 & \sigma_{c}^2 & \cdots & \sigma_{c}^2 + \sigma_u^2 \end{bmatrix} \tag{9.2}\]
The random effect estimator is:
\[ {\symbf{\hat \beta_{RE} = (\sum_{i=1}^N X_i' \hat \Omega^{-1} X_i)^{-1} (\sum_{i=1}^N X_i' \hat \Omega^{-1} y_i)}} \tag{9.3}\]
It is a special case of GLS. Before calculating \(\hat \beta_{RE}\), \(\sigma_{c}^2\) and \(\sigma_{u}^2\) need to be estimated. This is generally done by using residuals from within and between regressions (the Swamy-Arora method, the default in most software), or from pooled OLS.
Random effect methods assume that \(c_i\) be orthogonal to \({\symbf{x_{it}}}\). In many applications, the whole point of using panel data is to allow \(c_i\) be correlated with \({\symbf{x_{it}}}\). We need fixed-effect models in those cases.
For the same model as in the equation above, the fixed-effect methods try to eliminate \(c_i\) using fixed effects transformation, or “within transformation”. The FE transformation is to “de-mean” each observation by subtracting the group mean. Basically subtracting the group-mean equation below from the unit-level equation above:
\[ \bar y_i={{\symbf{\bar x_i \beta}}} + c_i + \bar u_i \tag{9.4}\]
where \(\bar y_i\), etc., means group means. For example, if we have 50 states with state level data of cigarette consumption across years, then groups means states, and we have 50 group means which are state level average cigarette consumption across years.
The reason for “demeaning” is to remove \(c_i\) from final estimation. The “demeaned” regression equation is the final regression used in estimation:
\[ (y_{it}-\bar y_i) = ({{\symbf{x}}}_{it}-{{\symbf{\bar x}}}_i)\beta + (u_{it}-\bar u_i). \tag{9.5}\]
The unit-specific effect \(c_i\) cancels because it is constant over \(t\) (\(c_i - \bar c_i = 0\)). Writing the within-transformed variables as \(\ddot y_{it}=y_{it}-\bar y_i\) and \(\ddot{{\symbf{x}}}_{it}={{\symbf{x}}}_{it}-{{\symbf{\bar x}}}_i\), OLS on the demeaned data gives the fixed-effects (within) estimator (here \({{\symbf{x}}}_{it}\) is a \(1\times k\) row vector, so \(\ddot{{\symbf{x}}}_{it}'\ddot{{\symbf{x}}}_{it}\) is the \(k\times k\) outer product)
\[ \hat\beta_{FE} = \left(\sum_i \sum_t \ddot{{\symbf{x}}}_{it}'\ddot{{\symbf{x}}}_{it}\right)^{-1} \sum_i \sum_t \ddot{{\symbf{x}}}_{it}'\ddot y_{it}. \tag{9.6}\]
This is consistent for \(\beta\) even when \(c_i\) is correlated with \({{\symbf{x}}}_{it}\), which is the key advantage of FE over random effects. Consistency still asks for the rest of the usual package: strict exogeneity of the idiosyncratic error conditional on the unit effect, \(E[u_{it} \mid c_i, {{\symbf{x}}}_{i1}, \dots, {{\symbf{x}}}_{iT}]=0\), enough within-unit variation for the demeaned cross-product to have full rank, and an asymptotic scheme – usually \(N \to \infty\) with \(T\) fixed. What FE buys is freedom from correlation between \(c_i\) and the regressors. It buys nothing at all against correlation between \(u_{it}\) and them. The price is that any time-invariant regressor (the \(z_i\) above) is also swept out by the demeaning and cannot be estimated. The usual variance estimator is \(\hat V(\hat\beta_{FE})=\hat\sigma_u^2(\sum_i\sum_t \ddot{{\symbf{x}}}_{it}'\ddot{{\symbf{x}}}_{it})^{-1}\) with \(\hat\sigma_u^2\) using degrees of freedom \(NT-N-k\); in practice one clusters by unit to allow serial correlation within \(i\).
Random effects is more efficient (it uses both within and between variation) but is consistent only if \(c_i\) is uncorrelated with \({{\symbf{x}}}_{it}\); fixed effects is consistent either way. The Hausman test compares the two estimators:
\[ H = (\hat\beta_{FE}-\hat\beta_{RE})'\,[\hat V(\hat\beta_{FE})-\hat V(\hat\beta_{RE})]^{-1}\,(\hat\beta_{FE}-\hat\beta_{RE}) \;\sim\; \chi^2_k \tag{9.7}\]
under the null that random effects is consistent. A large \(H\) (small \(p\)-value) favors fixed effects. In R this is phtest(fe, re) from the plm package; in Stata, hausman fe re.
Caveat: this classic form requires RE to be efficient. The formula above relies on \(\hat V(\hat\beta_{FE})-\hat V(\hat\beta_{RE})\) being positive semi-definite, which in turn requires the RE estimator to actually be the efficient estimator under the null – true only under the RE model’s homoscedasticity and no-serial-correlation assumptions. If you are using robust or cluster-robust covariance matrices (the usual practice when errors are heteroscedastic or serially correlated), RE is generally no longer efficient, the covariance difference can fail to be positive definite, and the classic Hausman test statistic can be negative or otherwise ill-behaved. In that case, use a regression-based Hausman test instead: augment the RE (or pooled) regression with the within-unit means of the time-varying regressors (Mundlak’s approach) or with the demeaned regressors themselves (Wooldridge’s regression-based test), and test the added coefficients jointly with a standard (robust) Wald test – this is valid under heteroscedasticity/clustering and reduces to the same idea as the classic Hausman test under its assumptions.
A practical question comes up whenever we hold a panel. Should we average each unit over time and run one regression on the \(N\) resulting rows, or keep all \(NT\) rows and cluster the standard errors by unit? Many people treat this as a trade-off and assume the long format must carry more information. It usually does not. The answer turns entirely on where the regressors live.
Take the case where every regressor is time-invariant, so the model is \(y_{it}={{\symbf{z}}}_i \gamma + c_i + u_{it}\). With a balanced panel, pooled OLS on the \(NT\) rows and OLS on the \(N\) unit means give the same \(\hat\gamma\). Not close — identical. Random effects returns the same number as well. Wooldridge (2010) states the reason plainly: because \(c_i\) is there, new observations within a unit carry no information about \(\gamma\) beyond how they move the unit average \(\bar y_i\). In effect we only have \(N\) useful pieces of information. Averaging discards nothing, because \({{\symbf{z}}}_i\) was already constant over \(t\) and the within variation was never available to \(\gamma\) in the first place.
What does differ is the degrees of freedom. The unit-mean regression uses \(N-k\); pooled OLS reports \(NT-k\), and that is the mistake. Moulton (1990) is the classic illustration, and Donald and Lang (2007) show that \(t_{N-k}\) is the reference distribution that keeps the size right here – an exact result resting on the distributional and design assumptions of the group-level regression, not a universal small-sample repair. With few clusters and heteroskedasticity the practical routes are a cluster-robust variance with a few-cluster correction, or a cluster bootstrap. Clustering by unit repairs the standard error when \(N\) is large, but it does not repair the degrees of freedom when \(N\) is small.
Now put a time-varying \({{\symbf{x}}}_{it}\) into the model. The two options stop agreeing, and not only for the coefficient on \({{\symbf{x}}}_{it}\) — every coefficient moves, the one on \({{\symbf{z}}}_i\) included. Averaging to unit means throws away all within variation, so what survives is the between estimator. Pooled OLS keeps both sources and forces a single coefficient on them, so it returns a variance-weighted average of the within and the between relationship. The two agree only when the within and between coefficients are equal, which is what the Hausman test is looking for in practice. Strictly the null is consistency of random effects – orthogonality of \(c_i\) and the regressor history – which implies equal within and between coefficients rather than being the same statement. Equality could hold under an alternative, and equality in one sample is not the null at all. So this is not a new problem. It is the FE-versus-RE question in different clothes.
| Situation | Collapsed vs long |
|---|---|
| All regressors time-invariant, balanced panel | Same coefficients; only the degrees of freedom differ |
| Any time-varying regressor | Coefficients generally differ, including those on \({{\symbf{z}}}_i\) |
| Unbalanced panel | Generally differ, since pooled OLS weights each unit by \(T_i\) |
The word to hold on to in the last two rows is generally. The differences come from the design, not from a theorem: a regressor orthogonal to the rest, or \(T_i\) variation that happens to leave the weighting unchanged, can leave particular coefficients coinciding. What you lose is the right to count on it.
The reconciliation is Mundlak’s device, the same one used above for the regression-based Hausman test. Add the unit means \({{\symbf{\bar x}}}_i\) to the long regression alongside \({{\symbf{x}}}_{it}\), and the within and between variation carry separate coefficients: \({{\symbf{x}}}_{it}\) picks up the within estimate, \({{\symbf{\bar x}}}_i\) picks up the difference between the between and within estimates, so the two coefficients sum to what the collapsed regression reports. That second coefficient is the Hausman contrast. With a single time-varying regressor its \(t\)-statistic is the Hausman test; with several, the test is the joint Wald or \(F\) test on all the mean coefficients together, not any one \(t\). Exact numerical agreement with the collapsed regression also depends on balance and on the weighting, so read it as the same comparison rather than an identity to check to the last digit. We keep the within variation and recover the between answer in one regression.
The useful way to remember it: collapse freely when every regressor is constant within the group, and never when one is not.
The section above is mostly a warning. The other side is worth seeing directly, so we simulate panels where we know the truth. We use fixest here; in Stata the same regressions are regress, xtreg, be and xtreg, fe.
Take 1000 units over 6 periods, one time-invariant regressor \(z_i\), and a true \(\gamma=2\).
set.seed(42)
N <- 1000; T <- 6
z <- rnorm(N)
bal <- data.frame(id = rep(1:N, each = T), t = rep(1:T, N),
z = rep(z, each = T), c = rep(rnorm(N), each = T))
bal$y <- 2 * bal$z + bal$c + rnorm(N * T)
agg <- aggregate(cbind(y, z) ~ id, data = bal, FUN = mean)
print(c(long = coef(feols(y ~ z, bal))[["z"]],
collapsed = coef(feols(y ~ z, agg))[["z"]]), digits = 12) long collapsed
1.99351677619 1.99351677619 The two agree to every digit printed. Now break the balance by dropping the later periods for units with high \(z_i\), so \(T_i\) varies, and varies with the regressor.
long collapsed
2.00682070731 2.01481052322 They no longer agree. Pooled OLS weights each unit by \(T_i\); the collapsed regression counts each unit once. Neither is wrong — they answer different questions, and we should know which one we asked. That \(T_i\) is a special case of a general rule, set out in the next section.
Now the case that matters. Let \(x_{it}\) vary over time and make \(c_i\) correlated with \(\bar x_i\), which is the situation fixed effects exists for. The truth is \(\beta=1\).
mu <- rnorm(N)
d <- data.frame(id = rep(1:N, each = T),
x = rep(mu, each = T) + rnorm(N * T),
c = rep(mu + rnorm(N), each = T))
d$y <- 1 * d$x + d$c + rnorm(N * T)
d$xbar <- ave(d$x, d$id)
dagg <- aggregate(cbind(y, x) ~ id, data = d, FUN = mean)
between <- feols(y ~ x, data = dagg)
pooled <- feols(y ~ x, data = d, cluster = ~id)
within <- feols(y ~ x | id, data = d)
mundlak <- feols(y ~ x + xbar, data = d, cluster = ~id)
pull <- function(m, v) c(coef(m)[[v]], se(m)[[v]])
tab <- rbind(`between (collapsed)` = pull(between, "x"),
`pooled OLS` = pull(pooled, "x"),
`within / FE` = pull(within, "x"),
`Mundlak: x` = pull(mundlak, "x"),
`Mundlak: xbar` = pull(mundlak, "xbar"))
colnames(tab) <- c("estimate", "std. error")
round(tab, 3) estimate std. error
between (collapsed) 1.845 0.035
pooled OLS 1.484 0.022
within / FE 0.995 0.014
Mundlak: x 0.995 0.014
Mundlak: xbar 0.850 0.036Read the table down. The collapsed regression returns about 1.85, badly biased, because it is the between estimator and \(\bar x_i\) carries \(c_i\) with it. Pooled OLS returns about 1.48 — closer, since some within variation is mixed in, but still inconsistent. The within estimator returns about 0.99, and only the long format can compute it: once we collapse there is nothing left to demean. Mundlak delivers both at once, exactly as described above — 0.99 on \(x_{it}\), 0.85 on \(\bar x_i\), and \(0.99+0.85\) recovers the between estimate.
Mundlak’s device is more than a test: it is an estimator in its own right. Wooldridge’s correlated random effects (CRE) framework generalizes the idea, allowing unit-level heterogeneity to depend on the time-averages of the regressors while retaining the RE structure. The full CRE treatment, including extensions to nonlinear models, is in the Blog Book.
So the argument for the long format is not precision. It is that demeaning, the Hausman comparison, unit-specific slopes, lagged dependent variables and anything else built on within variation stop existing once the panel is collapsed. Dynamic models are the extreme case, and they get their own chapter, Dynamic Panel Data.
The rule behind that \(T_i\), and behind the variance weighting in the between-versus-within comparison, is not specific to panels. It is set out in Interpreting OLS: Effect Weights and its companion, Outcome Weights.