3 GLS

3.1 Introduction

Since heteroscedasticity and serial correlation affect both linear and nonlinear regression models in the same way, there is no harm in limiting our attention to the simpler linear case.

\[ {\symbf{y=X\beta+u, \quad {E} (u)=0, \quad {E} (u u')=\Omega, }} \tag{3.1}\] where ${\symbf{\Omega}}$, the variance-covariance matrix of the error term, is a positive definite $n \times n$ matrix. If ${\symbf{\Omega}}$ is equal to $\sigma^2 {\symbf{I}}$, then it is just the linear regression model without heteroscedasticity and serial correlation. If ${\symbf{\Omega}}$ is diagonal with nonconstant diagonal elements, then the error terms are still uncorrelated, but there is heteroscedasticity. If ${\symbf{\Omega}}$ is not diagonal, then $u_i$ and $u_j$ are correlated. In next section, we obtain an efficient estimator for the vector $\beta$ by transforming the regression so that it satisfies the conditions of Gauss-Markov Theorem. This efficient estimator is called the Generalized Least Squares, or GLS, estimator.

3.2 The GLS Estimator

Since ${\symbf{\Omega}}$, the variance-covariance matrix of the error term, is a positive definite $n \times n$ matrix, there always exist full-rank $n \times n$ matrices (usually triangular) ${\symbf{\Psi}}$ such that \[ {\symbf{\Omega^{-1} =\Psi \Psi'. }} \tag{3.2}\]

Premultiplying (the equation above) by ${\symbf{\Psi'}}$ gives \[ {\symbf{\Psi'y=\Psi'X\beta+\Psi'u. }} \tag{3.3}\]

The OLS estimator from regression (the equation above) is \[ {\symbf{\hat \beta_{GLS}=(X'\Psi \Psi' X)^{-1}X'\Psi \Psi' y=(X'\Omega^{-1} X)^{-1}X'\Omega^{-1} y }} \tag{3.4}\]

The transformed error term has an identity variance-covariance matrix: \[ {\symbf{E (\Psi' u u' \Psi)=\Psi' {E} (u u') \Psi= \Psi' \Omega \Psi=I }} \tag{3.5}\]

Since the transformed model satisfies OLS assumptions, we have

\[ \begin{aligned} {{\mathrm{Var}}} {\symbf{(\hat \beta_{GLS}) }}&{\symbf{= (X' \Psi \Psi'X)^{-1}}}\\{\symbf{ }}&{\symbf{= (X' \Omega^{-1} X)^{-1} }}\end{aligned} \tag{3.6}\]

3.3 Weighted Least Squares

It is easy to obtain GLS estimates when the error terms are heteroscedastic but uncorrelated, which means ${\symbf{\Omega}}$ is diagonal. Let $\omega_t^2$ denote the $t$th diagonal element of ${\symbf{\Omega}}$. Then ${\symbf{\Psi}}$ can be chosen as the diagonal matrix with $t$th diagonal element $\omega_t^{-1}$. For a typical observation, regression can be written as \[ \omega_t^{-1}y_t=\omega_t^{-1} {{\symbf{x}}}_t' \boldsymbol{\beta}+ \omega_t^{-1} u_t. \tag{3.7}\] This is called weighted least squares, or WLS. It is worth keeping two notions of “weight” distinct. The transformed variables $y_t$ and ${\symbf{x_t'}}$ are scaled by $\omega_t^{-1}$; equivalently, the WLS objective weights each squared residual by the inverse variance $\omega_t^{-2}$ (the square of the scaling factor). Either way, observations whose error variance is large are given low weight, and observations for which it is small are given high weight.

3.3.1 Feasible Generalized Least Squares

In many cases it is reasonable to suppose that ${\symbf{\Omega}}$ depends in a known way on a vector of unknown parameters ${\symbf{\gamma}}$. If so, it may be possible to estimate $\mathbf{\gamma}$ consistently, so as to obtain ${\symbf{\Omega (\hat \gamma)}}$. This type of procedure is called feasible generalized least squares, or feasible GLS. But we’ll have to specify the error term as some function of some known variables.

A typical FGLS procedure has three steps. First, run OLS and obtain the residuals $\hat u_t$. Second, use those residuals to estimate the parameters $\gamma$ of the assumed error structure – for example:

Heteroscedasticity of known form, e.g. $\omega_t^2 = \exp({{\symbf{z}}}_t' \gamma)$: regress $\log(\hat u_t^2)$ on ${\symbf{z_t}}$ to estimate $\gamma$, then set $\hat\omega_t^2 = \exp({{\symbf{z}}}_t' \hat\gamma)$ and apply WLS as above. (The intercept of that auxiliary regression is inconsistent, because $E[\log(u_t^2/\omega_t^2)] \neq 0$ – it is about $-1.27$ for normal errors. This does not matter for WLS: a constant added to every $\log \hat\omega_t^2$ scales all weights by a common factor, which leaves $\hat\beta$ unchanged. It does matter if the fitted $\hat\omega_t^2$ are wanted as variance estimates in their own right.)
AR(1) serial correlation, $u_t = \rho u_{t-1} + \epsilon_t$: regress $\hat u_t$ on $\hat u_{t-1}$ to get $\hat\rho$, then quasi-difference each variable as $\tilde y_t = y_t - \hat\rho y_{t-1}$ for $t = 2, \dots, n$ (and similarly for $X$), so that the transformed error is approximately i.i.d.

The first observation is where Cochrane-Orcutt and Prais-Winsten part company, and the ${\symbf{\Psi}}$ framework above says which is right. For AR(1) errors ${\symbf{\Omega = \frac{\sigma_\epsilon^2}{1-\rho^2}\left[\rho^{|i-j|}\right]}}$, and its Cholesky factor makes the first row of ${\symbf{\Psi'}}$ equal to $\sqrt{1-\rho^2}$ times the first unit vector. Full GLS therefore keeps observation 1, rescaled as $\tilde y_1 = \sqrt{1-\hat\rho^2}\, y_1$; that is Prais-Winsten. Cochrane-Orcutt simply drops observation 1. Both are consistent and asymptotically equivalent, but Cochrane-Orcutt throws away a data point, and how much that costs takes some care. The direct weight on observation 1 is $\sqrt{1-\rho^2}$, which shrinks as $|\rho| \to 1$, so that row’s mechanical contribution falls rather than rises. What grows is its relative importance: quasi-differencing strips variation out of the remaining rows as well, and with trending or highly persistent regressors the single untransformed level observation ends up carrying a disproportionate share of the information about the level. So the loss is worth caring about in short samples, and with trending regressors when $\rho$ is near 1 – not because that row is weighted heavily, but because the others are weighted lightly. Prefer Prais-Winsten.

Third, run OLS on the transformed (weighted or quasi-differenced) data to get the FGLS estimator $\hat\beta_{FGLS}$. FGLS is consistent and asymptotically efficient under two requirements, not one: the assumed form of ${\symbf{\Omega}}$ must be correct, and $\hat\gamma$ must be consistent for $\gamma$ within that form – alongside the usual exogeneity, rank and moment conditions. A consistent $\hat\gamma$ inside a misspecified covariance model buys nothing. Either way FGLS is generally biased in finite samples, since $\hat\Omega$ depends on the first-stage OLS residuals.

3.4 Efficiency of GLS

3.4.1 The Gauss-Markov Theorem for GLS

The GLS estimator is the best linear unbiased estimator (BLUE) when ${\symbf{\Omega}}$ is known. To see this, consider any other linear unbiased estimator ${\symbf{\tilde\beta = Cy}}$ for some matrix ${\symbf{C}}$ satisfying ${\symbf{CX = I}}$. Write ${\symbf{C = (X'\Omega^{-1}X)^{-1}X'\Omega^{-1} + D}}$, where ${\symbf{D}}$ is the difference from the GLS weighting. Unbiasedness requires ${\symbf{CX = I}}$, which means ${\symbf{DX = 0}}$. The variance of ${\symbf{\tilde\beta}}$ is \[ \begin{aligned} {\symbf{\mathrm{Var}(\tilde\beta)}} &= {\symbf{C\,\Omega\, C'}} \\ &= {\symbf{(X'\Omega^{-1}X)^{-1} + D\,\Omega\, D'}}. \end{aligned} \tag{3.8}\]

The cross terms vanish because ${\symbf{DX = 0}}$ implies ${\symbf{D\,\Omega\,\Omega^{-1}X = DX = 0}}$, so ${\symbf{D\,\Omega\,\Omega^{-1}X(X'\Omega^{-1}X)^{-1} = 0}}$. Since ${\symbf{D\,\Omega\, D'}}$ is positive semidefinite, \[ {\symbf{\mathrm{Var}(\tilde\beta) - \mathrm{Var}(\hat\beta_{GLS}) = D\,\Omega\, D' \ge 0}}. \tag{3.9}\]

No linear unbiased estimator can have a smaller variance than GLS.

When ${\symbf{\Omega = \sigma^2 I}}$, GLS reduces to OLS, and this is the ordinary Gauss-Markov theorem: OLS is BLUE under homoscedasticity and no serial correlation.

3.4.2 GLS versus OLS with Robust Standard Errors

The Gauss-Markov result says GLS is more efficient than OLS when ${\symbf{\Omega}}$ is known. In practice ${\symbf{\Omega}}$ is never known, and we face a choice: estimate ${\symbf{\Omega}}$ and use FGLS, or use OLS with a sandwich variance estimator (White or Newey-West, as in the MLE chapter) that does not require specifying the form of ${\symbf{\Omega}}$.

Both approaches give consistent estimates of $\beta$, since OLS is consistent under the same conditions whether or not errors are heteroscedastic or serially correlated. The difference is efficiency. GLS exploits the structure of ${\symbf{\Omega}}$ to weight observations optimally; OLS ignores that structure and compensates afterward with a robust variance estimate.

The tradeoff turns on how well ${\symbf{\Omega}}$ can be modeled:

When the error structure is well understood — for example, heteroscedasticity that follows a known function of observables, or AR(1) serial correlation whose form is dictated by the data generating process — FGLS can deliver substantial efficiency gains. The efficiency improvement is largest when the heterogeneity in error variances is large, since GLS downweights noisy observations that OLS treats equally.

When the error structure is unknown or hard to model correctly, FGLS carries a specification risk. If ${\symbf{\hat\Omega}}$ is badly estimated, the FGLS estimator can be less efficient than OLS, and the FGLS standard errors can be misleading because they assume the estimated error structure is correct. OLS with robust standard errors avoids this risk: the point estimates are the same regardless of ${\symbf{\Omega}}$, and the sandwich variance is consistent under unknown heteroscedasticity, and under serial correlation weak enough to satisfy the HAC conditions – summable autocovariances and a bandwidth that grows slowly with the sample. “Robust” is not the same as robust to dependence of any form.

The modern default in applied work is to report OLS with robust standard errors, reserving GLS for settings where there is strong prior information about the error structure or where the efficiency gain is large enough to justify the specification risk. Panel data models are an important exception: the random effects estimator is a form of FGLS, and there the structure of ${\symbf{\Omega}}$ (an equicorrelated block for each unit) is implied by the model rather than estimated freely.

# GLS ## Introduction Since heteroscedasticity and serial correlation affect both linear and nonlinear regression models in the same way, there is no harm in limiting our attention to the simpler linear case. $$ {\symbf{y=X\beta+u, \quad {E} (u)=0, \quad {E} (u u')=\Omega, }}$$ {#eq-gls-1} where ${\symbf{\Omega}}$, the variance-covariance matrix of the error term, is a positive definite $n \times n$ matrix. If ${\symbf{\Omega}}$ is equal to $\sigma^2 {\symbf{I}}$, then it is just the linear regression model without heteroscedasticity and serial correlation. If ${\symbf{\Omega}}$ is diagonal with nonconstant diagonal elements, then the error terms are still uncorrelated, but there is heteroscedasticity. If ${\symbf{\Omega}}$ is not diagonal, then $u_i$ and $u_j$ are correlated. In next section, we obtain an efficient estimator for the vector $\beta$ by transforming the regression so that it satisfies the conditions of Gauss-Markov Theorem. This efficient estimator is called the Generalized Least Squares, or GLS, estimator. ## The GLS Estimator Since ${\symbf{\Omega}}$, the variance-covariance matrix of the error term, is a positive definite $n \times n$ matrix, there always exist full-rank $n \times n$ matrices (usually triangular) ${\symbf{\Psi}}$ such that $$ {\symbf{\Omega^{-1} =\Psi \Psi'. }}$$ {#eq-gls-2} Premultiplying (the equation above) by ${\symbf{\Psi'}}$ gives $$ {\symbf{\Psi'y=\Psi'X\beta+\Psi'u. }}$$ {#eq-gls-3} The OLS estimator from regression (the equation above) is $$ {\symbf{\hat \beta_{GLS}=(X'\Psi \Psi' X)^{-1}X'\Psi \Psi' y=(X'\Omega^{-1} X)^{-1}X'\Omega^{-1} y }}$$ {#eq-gls-4} The transformed error term has an identity variance-covariance matrix: $$ {\symbf{E (\Psi' u u' \Psi)=\Psi' {E} (u u') \Psi= \Psi' \Omega \Psi=I }}$$ {#eq-gls-5} Since the transformed model satisfies OLS assumptions, we have $$ \begin{aligned} {{\mathrm{Var}}} {\symbf{(\hat \beta_{GLS}) }}&{\symbf{= (X' \Psi \Psi'X)^{-1}}}\\{\symbf{ }}&{\symbf{= (X' \Omega^{-1} X)^{-1} }}\end{aligned} $$ {#eq-gls-6} ## Weighted Least Squares It is easy to obtain GLS estimates when the error terms are heteroscedastic but uncorrelated, which means ${\symbf{\Omega}}$ is diagonal. Let $\omega_t^2$ denote the $t$th diagonal element of ${\symbf{\Omega}}$. Then ${\symbf{\Psi}}$ can be chosen as the diagonal matrix with $t$th diagonal element $\omega_t^{-1}$. For a typical observation, regression can be written as $$ \omega_t^{-1}y_t=\omega_t^{-1} {{\symbf{x}}}_t' \boldsymbol{\beta}+ \omega_t^{-1} u_t. $$ {#eq-gls-7} This is called weighted least squares, or WLS. It is worth keeping two notions of "weight" distinct. The transformed variables $y_t$ and ${\symbf{x_t'}}$ are *scaled* by $\omega_t^{-1}$; equivalently, the WLS objective weights each squared residual by the inverse variance $\omega_t^{-2}$ (the square of the scaling factor). Either way, observations whose error variance is large are given low weight, and observations for which it is small are given high weight. ### Feasible Generalized Least Squares In many cases it is reasonable to suppose that ${\symbf{\Omega}}$ depends in a known way on a vector of unknown parameters ${\symbf{\gamma}}$. If so, it may be possible to estimate $\mathbf{\gamma}$ consistently, so as to obtain ${\symbf{\Omega (\hat \gamma)}}$. This type of procedure is called feasible generalized least squares, or feasible GLS. But we'll have to specify the error term as some function of some known variables. A typical FGLS procedure has three steps. First, run OLS and obtain the residuals $\hat u_t$. Second, use those residuals to estimate the parameters $\gamma$ of the assumed error structure -- for example: - **Heteroscedasticity of known form**, e.g. $\omega_t^2 = \exp({{\symbf{z}}}_t' \gamma)$: regress $\log(\hat u_t^2)$ on ${\symbf{z_t}}$ to estimate $\gamma$, then set $\hat\omega_t^2 = \exp({{\symbf{z}}}_t' \hat\gamma)$ and apply WLS as above. (The intercept of that auxiliary regression is inconsistent, because $E[\log(u_t^2/\omega_t^2)] \neq 0$ -- it is about $-1.27$ for normal errors. This does not matter for WLS: a constant added to every $\log \hat\omega_t^2$ scales all weights by a common factor, which leaves $\hat\beta$ unchanged. It does matter if the fitted $\hat\omega_t^2$ are wanted as variance estimates in their own right.) - **AR(1) serial correlation**, $u_t = \rho u_{t-1} + \epsilon_t$: regress $\hat u_t$ on $\hat u_{t-1}$ to get $\hat\rho$, then quasi-difference each variable as $\tilde y_t = y_t - \hat\rho y_{t-1}$ for $t = 2, \dots, n$ (and similarly for $X$), so that the transformed error is approximately i.i.d. The first observation is where **Cochrane-Orcutt** and **Prais-Winsten** part company, and the ${\symbf{\Psi}}$ framework above says which is right. For AR(1) errors ${\symbf{\Omega = \frac{\sigma_\epsilon^2}{1-\rho^2}\left[\rho^{|i-j|}\right]}}$, and its Cholesky factor makes the first row of ${\symbf{\Psi'}}$ equal to $\sqrt{1-\rho^2}$ times the first unit vector. Full GLS therefore keeps observation 1, rescaled as $\tilde y_1 = \sqrt{1-\hat\rho^2}\, y_1$; that is Prais-Winsten. Cochrane-Orcutt simply drops observation 1. Both are consistent and asymptotically equivalent, but Cochrane-Orcutt throws away a data point, and how much that costs takes some care. The direct weight on observation 1 is $\sqrt{1-\rho^2}$, which *shrinks* as $|\rho| \to 1$, so that row's mechanical contribution falls rather than rises. What grows is its relative importance: quasi-differencing strips variation out of the remaining rows as well, and with trending or highly persistent regressors the single untransformed level observation ends up carrying a disproportionate share of the information about the level. So the loss is worth caring about in short samples, and with trending regressors when $\rho$ is near 1 -- not because that row is weighted heavily, but because the others are weighted lightly. Prefer Prais-Winsten. Third, run OLS on the transformed (weighted or quasi-differenced) data to get the FGLS estimator $\hat\beta_{FGLS}$. FGLS is consistent and asymptotically efficient under two requirements, not one: the assumed *form* of ${\symbf{\Omega}}$ must be correct, and $\hat\gamma$ must be consistent for $\gamma$ within that form -- alongside the usual exogeneity, rank and moment conditions. A consistent $\hat\gamma$ inside a misspecified covariance model buys nothing. Either way FGLS is generally biased in finite samples, since $\hat\Omega$ depends on the first-stage OLS residuals. ## Efficiency of GLS ### The Gauss-Markov Theorem for GLS The GLS estimator is the best linear unbiased estimator (BLUE) when ${\symbf{\Omega}}$ is known. To see this, consider any other linear unbiased estimator ${\symbf{\tilde\beta = Cy}}$ for some matrix ${\symbf{C}}$ satisfying ${\symbf{CX = I}}$. Write ${\symbf{C = (X'\Omega^{-1}X)^{-1}X'\Omega^{-1} + D}}$, where ${\symbf{D}}$ is the difference from the GLS weighting. Unbiasedness requires ${\symbf{CX = I}}$, which means ${\symbf{DX = 0}}$. The variance of ${\symbf{\tilde\beta}}$ is $$ \begin{aligned} {\symbf{\mathrm{Var}(\tilde\beta)}} &= {\symbf{C\,\Omega\, C'}} \\ &= {\symbf{(X'\Omega^{-1}X)^{-1} + D\,\Omega\, D'}}. \end{aligned} $$ {#eq-gls-gm} The cross terms vanish because ${\symbf{DX = 0}}$ implies ${\symbf{D\,\Omega\,\Omega^{-1}X = DX = 0}}$, so ${\symbf{D\,\Omega\,\Omega^{-1}X(X'\Omega^{-1}X)^{-1} = 0}}$. Since ${\symbf{D\,\Omega\, D'}}$ is positive semidefinite, $$ {\symbf{\mathrm{Var}(\tilde\beta) - \mathrm{Var}(\hat\beta_{GLS}) = D\,\Omega\, D' \ge 0}}. $$ {#eq-gls-gm2} No linear unbiased estimator can have a smaller variance than GLS. When ${\symbf{\Omega = \sigma^2 I}}$, GLS reduces to OLS, and this is the ordinary Gauss-Markov theorem: OLS is BLUE under homoscedasticity and no serial correlation. ### GLS versus OLS with Robust Standard Errors The Gauss-Markov result says GLS is more efficient than OLS when ${\symbf{\Omega}}$ is known. In practice ${\symbf{\Omega}}$ is never known, and we face a choice: estimate ${\symbf{\Omega}}$ and use FGLS, or use OLS with a sandwich variance estimator (White or Newey-West, as in the [MLE chapter](mle.qmd)) that does not require specifying the form of ${\symbf{\Omega}}$. Both approaches give consistent estimates of $\beta$, since OLS is consistent under the same conditions whether or not errors are heteroscedastic or serially correlated. The difference is efficiency. GLS exploits the structure of ${\symbf{\Omega}}$ to weight observations optimally; OLS ignores that structure and compensates afterward with a robust variance estimate. The tradeoff turns on how well ${\symbf{\Omega}}$ can be modeled: When the error structure is well understood --- for example, heteroscedasticity that follows a known function of observables, or AR(1) serial correlation whose form is dictated by the data generating process --- FGLS can deliver substantial efficiency gains. The efficiency improvement is largest when the heterogeneity in error variances is large, since GLS downweights noisy observations that OLS treats equally. When the error structure is unknown or hard to model correctly, FGLS carries a specification risk. If ${\symbf{\hat\Omega}}$ is badly estimated, the FGLS estimator can be less efficient than OLS, and the FGLS standard errors can be misleading because they assume the estimated error structure is correct. OLS with robust standard errors avoids this risk: the point estimates are the same regardless of ${\symbf{\Omega}}$, and the sandwich variance is consistent under unknown heteroscedasticity, and under serial correlation weak enough to satisfy the HAC conditions -- summable autocovariances and a bandwidth that grows slowly with the sample. "Robust" is not the same as robust to dependence of any form. The modern default in applied work is to report OLS with robust standard errors, reserving GLS for settings where there is strong prior information about the error structure or where the efficiency gain is large enough to justify the specification risk. Panel data models are an important exception: the random effects estimator is a form of FGLS, and there the structure of ${\symbf{\Omega}}$ (an equicorrelated block for each unit) is implied by the model rather than estimated freely.