3 GLS

3.1 Introduction

Since heteroscedasticity and serial correlation affect both linear and nonlinear regression models in the same way, there is no harm in limiting our attention to the simpler linear case.

\[ \bf y=X\beta+u, \quad {\rm E} (u)=0, \quad {\rm E} (u u')=\Omega, \] where $\bf \Omega$, the variance-covariance matrix of the error term, is a positive definite $n \times n$ matrix. If $\bf \Omega$ is equal to $\sigma^2 \bf I$, then it is just the linear regression model without heteroscedasticity and serial correlation. If $\bf \Omega$ is diagonal with nonconstant diagonal elements, then the error terms are still uncorrelated, but there is heteroscedasticity. If $\bf \Omega$ is not diagonal, then $u_i$ and $u_j$ are correlated. In next section, we obtain an efficient estimator for the vector $\beta$ by transforming the regression so that it satisfies the conditions of Gauss-Markov Theorem. This efficient estimator is called the Generalized Least Squares, or GLS, estimator.

3.2 The GLS Estimator

Since $\bf \Omega$, the variance-covariance matrix of the error term, is a positive definite $n \times n$ matrix, there always exist full-rank $n \times n$ matrices (usually triangular) $\bf \Psi$ such that \[ \bf \Omega^{-1} =\Psi \Psi'. \]

Premultiplying (the equation above) by $\bf \Psi'$ gives \[ \bf \Psi'y=\Psi'X\beta+\Psi'u. \]

The OLS estimator from regression (the equation above) is \[ \bf \hat \beta_{GLS}=(X'\Psi \Psi' X)^{-1}X'\Psi \Psi' y=(X'\Omega^{-1} X)^{-1}X'\Omega^{-1} y \]

The transformed error term has an identity variance-covariance matrix: \[ \rm E \bf (\Psi' u u' \Psi)=\Psi' {\rm E} (u u') \Psi= \Psi' \Omega \Psi=I \]

Since the transformed model satisfies OLS assumptions, we have

\[ \begin{aligned} {\rm Var} \bf (\hat \beta_{GLS}) &= \bf (X' \Psi \Psi'X)^{-1}\\ &= \bf (X' \Omega^{-1} X)^{-1} \end{aligned} \]

3.3 Weighted Least Squares

It is easy to obtain GLS estimates when the error terms are heteroscedastic but uncorrelated, which means $\bf \Omega$ is diagonal. Let $\omega_t^2$ denote the $t$th diagonal element of $\bf \Omega$. Then $\bf \Psi$ can be chosen as the diagonal matrix with $t$th diagonal element $\omega_t^{-1}$. For a typical observation, regression can be written as \[ \omega_t^{-1}y_t=\omega_t^{-1} {\bf x}_t' \boldsymbol{\beta}+ \omega_t^{-1} u_t. \] This is called weighted least squares, or WLS. It is worth keeping two notions of “weight” distinct. The transformed variables $y_t$ and $\bf x_t'$ are scaled by $\omega_t^{-1}$; equivalently, the WLS objective weights each squared residual by the inverse variance $\omega_t^{-2}$ (the square of the scaling factor). Either way, observations whose error variance is large are given low weight, and observations for which it is small are given high weight.

3.3.1 Feasible Generalized Least Squares

In many cases it is reasonable to suppose that $\bf \Omega$ depends in a known way on a vector of unknown parameters $\bf \gamma$. If so, it may be possible to estimate $\mathbf{\gamma}$ consistently, so as to obtain $\bf \Omega (\hat \gamma)$. This type of procedure is called feasible generalized least squares, or feasible GLS. But we’ll have to specify the error term as some function of some known variables.

A typical FGLS procedure has three steps. First, run OLS and obtain the residuals $\hat u_t$. Second, use those residuals to estimate the parameters $\gamma$ of the assumed error structure – for example:

Heteroscedasticity of known form, e.g. $\omega_t^2 = \exp({\bf z}_t' \gamma)$: regress $\log(\hat u_t^2)$ on $\bf z_t$ to estimate $\gamma$, then set $\hat\omega_t^2 = \exp({\bf z}_t' \hat\gamma)$ and apply WLS as above.
AR(1) serial correlation, $u_t = \rho u_{t-1} + \epsilon_t$: regress $\hat u_t$ on $\hat u_{t-1}$ to get $\hat\rho$, then apply the Cochrane-Orcutt (or Prais-Winsten) transformation – quasi-differencing each variable as $\tilde y_t = y_t - \hat\rho y_{t-1}$ (and similarly for $X$) – so that the transformed error is approximately i.i.d.

Third, run OLS on the transformed (weighted or quasi-differenced) data to get the FGLS estimator $\hat\beta_{FGLS}$. FGLS is consistent and asymptotically efficient as long as $\hat\gamma$ is a consistent estimator of $\gamma$, even though it is generally biased in finite samples (since $\hat\Omega$ depends on the first-stage OLS residuals).

# GLS ## Introduction Since heteroscedasticity and serial correlation affect both linear and nonlinear regression models in the same way, there is no harm in limiting our attention to the simpler linear case. $$ \bf y=X\beta+u, \quad {\rm E} (u)=0, \quad {\rm E} (u u')=\Omega, $$ where $\bf \Omega$, the variance-covariance matrix of the error term, is a positive definite $n \times n$ matrix. If $\bf \Omega$ is equal to $\sigma^2 \bf I$, then it is just the linear regression model without heteroscedasticity and serial correlation. If $\bf \Omega$ is diagonal with nonconstant diagonal elements, then the error terms are still uncorrelated, but there is heteroscedasticity. If $\bf \Omega$ is not diagonal, then $u_i$ and $u_j$ are correlated. In next section, we obtain an efficient estimator for the vector $\beta$ by transforming the regression so that it satisfies the conditions of Gauss-Markov Theorem. This efficient estimator is called the Generalized Least Squares, or GLS, estimator. ## The GLS Estimator Since $\bf \Omega$, the variance-covariance matrix of the error term, is a positive definite $n \times n$ matrix, there always exist full-rank $n \times n$ matrices (usually triangular) $\bf \Psi$ such that $$ \bf \Omega^{-1} =\Psi \Psi'. $$ Premultiplying (the equation above) by $\bf \Psi'$ gives $$ \bf \Psi'y=\Psi'X\beta+\Psi'u. $$ The OLS estimator from regression (the equation above) is $$ \bf \hat \beta_{GLS}=(X'\Psi \Psi' X)^{-1}X'\Psi \Psi' y=(X'\Omega^{-1} X)^{-1}X'\Omega^{-1} y $$ The transformed error term has an identity variance-covariance matrix: $$ \rm E \bf (\Psi' u u' \Psi)=\Psi' {\rm E} (u u') \Psi= \Psi' \Omega \Psi=I $$ Since the transformed model satisfies OLS assumptions, we have $$ \begin{aligned} {\rm Var} \bf (\hat \beta_{GLS}) &= \bf (X' \Psi \Psi'X)^{-1}\\ &= \bf (X' \Omega^{-1} X)^{-1} \end{aligned} $$ ## Weighted Least Squares It is easy to obtain GLS estimates when the error terms are heteroscedastic but uncorrelated, which means $\bf \Omega$ is diagonal. Let $\omega_t^2$ denote the $t$th diagonal element of $\bf \Omega$. Then $\bf \Psi$ can be chosen as the diagonal matrix with $t$th diagonal element $\omega_t^{-1}$. For a typical observation, regression can be written as $$ \omega_t^{-1}y_t=\omega_t^{-1} {\bf x}_t' \boldsymbol{\beta}+ \omega_t^{-1} u_t. $$ This is called weighted least squares, or WLS. It is worth keeping two notions of "weight" distinct. The transformed variables $y_t$ and $\bf x_t'$ are *scaled* by $\omega_t^{-1}$; equivalently, the WLS objective weights each squared residual by the inverse variance $\omega_t^{-2}$ (the square of the scaling factor). Either way, observations whose error variance is large are given low weight, and observations for which it is small are given high weight. ### Feasible Generalized Least Squares In many cases it is reasonable to suppose that $\bf \Omega$ depends in a known way on a vector of unknown parameters $\bf \gamma$. If so, it may be possible to estimate $\mathbf{\gamma}$ consistently, so as to obtain $\bf \Omega (\hat \gamma)$. This type of procedure is called feasible generalized least squares, or feasible GLS. But we'll have to specify the error term as some function of some known variables. A typical FGLS procedure has three steps. First, run OLS and obtain the residuals $\hat u_t$. Second, use those residuals to estimate the parameters $\gamma$ of the assumed error structure -- for example: - **Heteroscedasticity of known form**, e.g. $\omega_t^2 = \exp({\bf z}_t' \gamma)$: regress $\log(\hat u_t^2)$ on $\bf z_t$ to estimate $\gamma$, then set $\hat\omega_t^2 = \exp({\bf z}_t' \hat\gamma)$ and apply WLS as above. - **AR(1) serial correlation**, $u_t = \rho u_{t-1} + \epsilon_t$: regress $\hat u_t$ on $\hat u_{t-1}$ to get $\hat\rho$, then apply the Cochrane-Orcutt (or Prais-Winsten) transformation -- quasi-differencing each variable as $\tilde y_t = y_t - \hat\rho y_{t-1}$ (and similarly for $X$) -- so that the transformed error is approximately i.i.d. Third, run OLS on the transformed (weighted or quasi-differenced) data to get the FGLS estimator $\hat\beta_{FGLS}$. FGLS is consistent and asymptotically efficient as long as $\hat\gamma$ is a consistent estimator of $\gamma$, even though it is generally biased in finite samples (since $\hat\Omega$ depends on the first-stage OLS residuals).