7 Discrete and Limited Dependent Variables

7.1 Binary Response Models

7.1.1 Probit and Logit

Let $P_t$ denote the probability that $y_t=1$ conditional on the information set $\Omega$, which consists of exogenous and predetermined variables. A binary response model serves to model this conditional expectation. Since the values are 0 or 1, it is clear that $P_t$ is also the expectation of $y_t$ conditional on $\Omega_t$: \[ P_t \equiv \mbox{Pr}(y_t=1 | \Omega_t)=\mbox{E}(y_t | \Omega_t). \tag{7.1}\]

Here $X_t$ is a set of regressors. A linear probability model would set $P_t = X_t \beta$ directly, which requires the fitted values to stay in $[0,1]$ – an awkward constraint, since the linear index $X_t \beta$ is unbounded. Logit and probit avoid this problem by modeling $P_t = F(X_t \beta)$, mapping the unbounded index through a CDF-valued transformation into $[0,1]$.

We ensure that $0 \le P_t \le 1$ by specifying that \[ P_t \equiv \mbox{Pr}(y_t=1 | \Omega_t)=F({\symbf{X_t \beta). }} \tag{7.2}\] $F(x)$ is a transformation function, which has the same characteristics as the CDF of a probability distribution.

Two popular choices of $F(x)$ are Gaussian (probit) and Logistic (logit).

The less familiar logistic function is \[ \Lambda(x)=\frac{e^x}{1+e^x} \tag{7.3}\]

The logit model is most easily derived by assuming that \[ \log (\frac{P_t}{1-P_t})={\symbf{X_t \beta }} \tag{7.4}\] which says the logarithm of the odds (the ratio of the two probabilities) is equal to ${\symbf{X_t \beta}}$. Therefore, \[ P_t =\frac{\exp({{\symbf{X}}}_t \beta)}{1+\exp({{\symbf{X}}}_t \beta)}=\Lambda({{\symbf{X}}}_t \beta) \tag{7.5}\]

7.1.2 MLE for binary data

The likelihood for an observation $t$ is the probability that $y_t=1$ if $y_t=1$, or the probability that $y_t=0$ if $y_t=0$. The logarithm of the appropriate probability is then the contribution to the loglikelihood made by observation $t$. Therefore, if ${\symbf{y}}$ is an n-vector with typical element $y_t$, the loglikelihood function for ${\symbf{y}}$ can be written as \[ {\ell}({{\symbf{y, \beta}}})=\sum_{t=1}^n(y_t \log F({{\symbf{X_t \beta}}})+(1-y_t) \log(1- F({{\symbf{X_t \beta}}}))) \tag{7.6}\]

For the logit and probit models, this function is globally concave with respect to $\beta$, so there are no local maxima to worry about. Concavity is not the whole story, though. For the MLE to exist and be unique and finite we also need ${\symbf{X}}$ to have full column rank, and the data must not be separable. If some linear combination of the regressors predicts the outcome perfectly – complete or quasi-complete separation – the likelihood keeps rising along that direction and the maximum is attained at no finite $\beta$, concave or not. Software reports this as coefficients diverging or standard errors blowing up. With full rank and no separation, the first-order conditions, or likelihood equations, uniquely define the MLE estimator $\hat \beta$. These likelihood equations can be written as \[ \sum_{t=1}^n \frac{(y_t-F({{\symbf{X_t \beta}}}))f({{\symbf{X_t \beta}}})x_{ti}}{F({{\symbf{X_t \beta}}})(1- F({{\symbf{X_t \beta}}}))}=0, \ i=1, \dots, k. \tag{7.7}\] where $f=F'$ is the density (e.g. $\phi$ for probit). For logit, $f=F(1-F)$, so the numerator cancels the denominator and the score reduces to $\sum_t (y_t-F)x_{ti}=0$.

Newton’s Method can be used to find $\hat \beta$.

7.2 Models for More than Two Discrete Responses

7.2.1 The Ordered Probit

Ordered Probit can be easily derived from a latent variable model. \[ y_t^0={{\symbf{X}}}_t \beta+u_t, \ u_t \sim NID(0,1) \tag{7.8}\]

Suppose we observe $y_t$ with three values.

\[ \begin{cases} y_t = 0 & \mbox{if} \ y_t^0 < \gamma_1; \\ y_t = 1 & \mbox{if} \ \gamma_1 \leq y_t^0 < \gamma_2; \\ y_t = 2 & \mbox{if} \ y_t^0 \geq \gamma_2. \end{cases} \tag{7.9}\]

Therefore, \[ \begin{aligned} \mbox{Pr}(y_t=0) &= \mbox{Pr}(y_t^0 < \gamma_1)=\mbox{Pr}({{\symbf{X}}}_t \beta + u_t < \gamma_1) \\ &= \mbox{Pr}(u_t < \gamma_1 - {{\symbf{X}}}_t \beta)=\Phi ( \gamma_1 - {{\symbf{X}}}_t \beta) \end{aligned} \tag{7.10}\]

Similarly, \[ \mbox{Pr}(y_t=2) = 1 - \Phi ( \gamma_2 - {{\symbf{X}}}_t \beta) \tag{7.11}\] \[ \mbox{Pr}(y_t=1) = \Phi ( \gamma_2- {{\symbf{X}}}_t \beta ) - \Phi ( \gamma_1- {{\symbf{X}}}_t \beta ) \tag{7.12}\]

These probabilities depend solely on the value of the index function and on the two threshold parameters.

Two normalizations are needed before any of this can be estimated, and they are easy to pass over. First, the location: an intercept in ${{\symbf{X}}}_t \beta$ and both thresholds cannot be identified together, since adding a constant to the index and the same constant to $\gamma_1$ and $\gamma_2$ leaves every probability unchanged. Software resolves this by dropping the intercept or by fixing $\gamma_1 = 0$. Second, the scale: that is what setting $\mbox{Var}(u_t) = 1$ above does, since multiplying the index, the thresholds and $u_t$ by any positive constant is likewise unobservable. The analogous restriction for the multinomial logit below is $\beta^0 = 0$.

The loglikelihood function is \[ \ell (\beta, \gamma_1, \gamma_2) = \sum_{y_t=0} \log(\Phi (\gamma_1 - {{\symbf{X}}}_t \beta)) + \sum_{y_t=2} \log(\Phi ( {{\symbf{X}}}_t \beta - \gamma_2)) + \sum_{y_t=1} \log(\Phi ( \gamma_2 - {{\symbf{X}}}_t \beta ) -\Phi ( \gamma_1 - {{\symbf{X}}}_t \beta )) \tag{7.13}\]

7.2.2 The Multinomial Logit

When responses are unordered, two popular choices are multinomial logit and conditional logit. They are easy to conflate because they share the same softmax form, but they differ in what varies across the $J+1$ alternatives $l=0,\dots,J$: in multinomial logit the regressors are individual-specific and only the coefficients vary by alternative; in conditional logit it’s the reverse.

Multinomial logit. Here the regressor vector ${\symbf{X_t}}$ (individual-specific characteristics, e.g. age, income) is the same for every alternative, and it is the coefficient vector that is alternative-specific:

\[ \mbox{Pr}(y_t=l)=\frac{\exp({{\symbf{X}}}_t \beta^l)}{\sum_{j=0}^J\exp({{\symbf{X}}}_t \beta^j)} \quad \mbox{for} \ l=0, \dots, J, \tag{7.14}\]

Here ${\symbf{X_t}}$ is a row vector of length $k$ of individual-specific covariates, and $\beta^l$ is a $k$-vector of alternative-specific parameters. For identification one category is taken as the base with $\beta^0=0$ (so its numerator is $1$).

Conditional logit. Here it is the regressors that vary by alternative (e.g. the price or characteristics of choice $j$ itself), while the coefficient vector is common across alternatives:

\[ \mbox{Pr}(y_t=l)=\frac{\exp({{\symbf{W}}}_{tl} \beta)}{\sum_{j=0}^J\exp({{\symbf{W}}}_{tj} \beta)} \quad \mbox{for} \ l=0, \dots, J, \tag{7.15}\]

Here ${{\symbf{W}}}_{tj}$ is a row vector of length $k$ of alternative-$j$-specific variables, and $\beta$ is a single $k$-vector of parameters shared across all alternatives $j=0,\dots,J$.

For binary and multinomial models alike, the coefficients $\beta$ do not directly measure the effect of a unit change in $X$ on the probability. That effect depends on where in the distribution the change occurs. The average marginal effect (AME) — averaging the partial effect over the observed distribution of covariates — is the standard summary. The full treatment of AME vs. marginal effects at the mean, including fixed-effects models, is in the Blog Book.

Writing both choice-varying regressors ${{\symbf{W}}}_{tl}$ and choice-specific coefficients $\beta^l$ into the same formula, as some treatments do, conflates the two models. It is worth being precise about what that costs, because the usual complaint – that such a specification is not identified – is not true. Choice probabilities depend only on utility differences, so $\Pr(y_t=l)$ is governed by ${{\symbf{W}}}_{tj}\beta^j - {{\symbf{W}}}_{tl}\beta^l$; when the ${{\symbf{W}}}_{tj}$ vary independently across alternatives, those differences pin down each $\beta^j$ separately, with no normalization required. It is a perfectly estimable model – it is what Stata’s asclogit fits when alternative-specific coefficients are requested.

What the ambiguous formula really costs is clarity about which normalization is in force. Individual-specific regressors require one alternative’s coefficients to be fixed ($\beta^0=0$), since only differences across alternatives are identified; alternative-specific regressors entering with a common coefficient require no such restriction. One caveat on “no normalization required” above: it concerns the slope coefficients $\beta^j$. If the specification also carries alternative-specific intercepts, those are identified only up to a location shift and one of them still has to be fixed. A formula that does not say which of the two it means leaves the reader unable to tell what has been normalized, and therefore unable to interpret the coefficients.

7.2.3 Independence of Irrelevant Alternatives

Both logit forms carry a restriction that comes from the softmax itself. Take the ratio of two choice probabilities. With individual-specific regressors,

\[ \frac{\Pr(y_t = j)}{\Pr(y_t = l)} = \exp \bigl( {{\symbf{X}}}_t (\beta^j - \beta^l) \bigr), \tag{7.16}\]

and with alternative-specific regressors and a common coefficient the ratio is $\exp \bigl( ({{\symbf{W}}}_{tj} - {{\symbf{W}}}_{tl}) \beta \bigr)$. Either way everything to do with the other alternatives cancels. The relative odds of $j$ against $l$ do not depend on what else is on the menu, or on the attributes of any third alternative. That is independence of irrelevant alternatives, or IIA.

Sometimes it is harmless. Often it is not. The standard illustration is travel mode. Suppose commuters split evenly between a car and a red bus, so the odds are 1:1 and each share is $1/2$. Now add a blue bus, identical to the red one in everything but colour. IIA holds the car-to-red-bus odds at 1:1, and the two buses are interchangeable, so the model predicts three shares of $1/3$. What we would expect is that the blue bus draws almost entirely from the red bus, leaving the car near $1/2$. The model has no way to know that the two buses are close substitutes, because the softmax gives it no way to express that.

There are two standard escapes. Nested logit groups alternatives into nests – the two buses in one, the car in another – and allows correlation within a nest, so IIA holds inside nests but not across them. Mixed logit, also called random-coefficients logit, lets $\beta$ vary across individuals; shared taste parameters then induce correlation across alternatives and IIA goes away, at the cost of simulation-based estimation. Which one to reach for depends on whether the substitution pattern you are worried about can be described by a grouping.

# Discrete and Limited Dependent Variables ## Binary Response Models ### Probit and Logit Let $P_t$ denote the probability that $y_t=1$ conditional on the information set $\Omega$, which consists of exogenous and predetermined variables. A binary response model serves to model this conditional expectation. Since the values are 0 or 1, it is clear that $P_t$ is also the expectation of $y_t$ conditional on $\Omega_t$: $$ P_t \equiv \mbox{Pr}(y_t=1 | \Omega_t)=\mbox{E}(y_t | \Omega_t). $$ {#eq-discrete-1} Here $X_t$ is a set of regressors. A linear probability model would set $P_t = X_t \beta$ directly, which requires the fitted values to stay in $[0,1]$ -- an awkward constraint, since the linear index $X_t \beta$ is unbounded. Logit and probit avoid this problem by modeling $P_t = F(X_t \beta)$, mapping the unbounded index through a CDF-valued transformation into $[0,1]$. We ensure that $0 \le P_t \le 1$ by specifying that $$ P_t \equiv \mbox{Pr}(y_t=1 | \Omega_t)=F({\symbf{X_t \beta). }}$$ {#eq-discrete-2} $F(x)$ is a transformation function, which has the same characteristics as the CDF of a probability distribution. Two popular choices of $F(x)$ are Gaussian (probit) and Logistic (logit). The less familiar logistic function is $$ \Lambda(x)=\frac{e^x}{1+e^x} $$ {#eq-discrete-3} The logit model is most easily derived by assuming that $$ \log (\frac{P_t}{1-P_t})={\symbf{X_t \beta }}$$ {#eq-discrete-4} which says the logarithm of the odds (the ratio of the two probabilities) is equal to ${\symbf{X_t \beta}}$. Therefore, $$ P_t =\frac{\exp({{\symbf{X}}}_t \beta)}{1+\exp({{\symbf{X}}}_t \beta)}=\Lambda({{\symbf{X}}}_t \beta) $$ {#eq-discrete-5} ### MLE for binary data The likelihood for an observation $t$ is the probability that $y_t=1$ if $y_t=1$, or the probability that $y_t=0$ if $y_t=0$. The logarithm of the appropriate probability is then the contribution to the loglikelihood made by observation $t$. Therefore, if ${\symbf{y}}$ is an n-vector with typical element $y_t$, the loglikelihood function for ${\symbf{y}}$ can be written as $$ {\ell}({{\symbf{y, \beta}}})=\sum_{t=1}^n(y_t \log F({{\symbf{X_t \beta}}})+(1-y_t) \log(1- F({{\symbf{X_t \beta}}}))) $$ {#eq-discrete-6} For the logit and probit models, this function is globally concave with respect to $\beta$, so there are no local maxima to worry about. Concavity is not the whole story, though. For the MLE to exist and be unique and *finite* we also need ${\symbf{X}}$ to have full column rank, and the data must not be separable. If some linear combination of the regressors predicts the outcome perfectly -- complete or quasi-complete separation -- the likelihood keeps rising along that direction and the maximum is attained at no finite $\beta$, concave or not. Software reports this as coefficients diverging or standard errors blowing up. With full rank and no separation, the first-order conditions, or likelihood equations, uniquely define the MLE estimator $\hat \beta$. These likelihood equations can be written as $$ \sum_{t=1}^n \frac{(y_t-F({{\symbf{X_t \beta}}}))f({{\symbf{X_t \beta}}})x_{ti}}{F({{\symbf{X_t \beta}}})(1- F({{\symbf{X_t \beta}}}))}=0, \ i=1, \dots, k. $$ {#eq-discrete-7} where $f=F'$ is the density (e.g. $\phi$ for probit). For logit, $f=F(1-F)$, so the numerator cancels the denominator and the score reduces to $\sum_t (y_t-F)x_{ti}=0$. Newton's Method can be used to find $\hat \beta$. ## Models for More than Two Discrete Responses ### The Ordered Probit Ordered Probit can be easily derived from a latent variable model. $$ y_t^0={{\symbf{X}}}_t \beta+u_t, \ u_t \sim NID(0,1) $$ {#eq-discrete-8} Suppose we observe $y_t$ with three values. $$ \begin{cases} y_t = 0 & \mbox{if} \ y_t^0 < \gamma_1; \\ y_t = 1 & \mbox{if} \ \gamma_1 \leq y_t^0 < \gamma_2; \\ y_t = 2 & \mbox{if} \ y_t^0 \geq \gamma_2. \end{cases} $$ {#eq-discrete-9} Therefore, $$ \begin{aligned} \mbox{Pr}(y_t=0) &= \mbox{Pr}(y_t^0 < \gamma_1)=\mbox{Pr}({{\symbf{X}}}_t \beta + u_t < \gamma_1) \\ &= \mbox{Pr}(u_t < \gamma_1 - {{\symbf{X}}}_t \beta)=\Phi ( \gamma_1 - {{\symbf{X}}}_t \beta) \end{aligned} $$ {#eq-discrete-10} Similarly, $$ \mbox{Pr}(y_t=2) = 1 - \Phi ( \gamma_2 - {{\symbf{X}}}_t \beta) $$ {#eq-discrete-11} $$ \mbox{Pr}(y_t=1) = \Phi ( \gamma_2- {{\symbf{X}}}_t \beta ) - \Phi ( \gamma_1- {{\symbf{X}}}_t \beta ) $$ {#eq-discrete-12} These probabilities depend solely on the value of the index function and on the two threshold parameters. Two normalizations are needed before any of this can be estimated, and they are easy to pass over. First, the *location*: an intercept in ${{\symbf{X}}}_t \beta$ and both thresholds cannot be identified together, since adding a constant to the index and the same constant to $\gamma_1$ and $\gamma_2$ leaves every probability unchanged. Software resolves this by dropping the intercept or by fixing $\gamma_1 = 0$. Second, the *scale*: that is what setting $\mbox{Var}(u_t) = 1$ above does, since multiplying the index, the thresholds and $u_t$ by any positive constant is likewise unobservable. The analogous restriction for the multinomial logit below is $\beta^0 = 0$. The loglikelihood function is $$ \ell (\beta, \gamma_1, \gamma_2) = \sum_{y_t=0} \log(\Phi (\gamma_1 - {{\symbf{X}}}_t \beta)) + \sum_{y_t=2} \log(\Phi ( {{\symbf{X}}}_t \beta - \gamma_2)) + \sum_{y_t=1} \log(\Phi ( \gamma_2 - {{\symbf{X}}}_t \beta ) -\Phi ( \gamma_1 - {{\symbf{X}}}_t \beta )) $$ {#eq-discrete-13} ### The Multinomial Logit When responses are unordered, two popular choices are multinomial logit and conditional logit. They are easy to conflate because they share the same softmax form, but they differ in what varies across the $J+1$ alternatives $l=0,\dots,J$: in multinomial logit the *regressors* are individual-specific and only the *coefficients* vary by alternative; in conditional logit it's the reverse. **Multinomial logit.** Here the regressor vector ${\symbf{X_t}}$ (individual-specific characteristics, e.g. age, income) is the same for every alternative, and it is the coefficient vector that is alternative-specific: $$ \mbox{Pr}(y_t=l)=\frac{\exp({{\symbf{X}}}_t \beta^l)}{\sum_{j=0}^J\exp({{\symbf{X}}}_t \beta^j)} \quad \mbox{for} \ l=0, \dots, J, $$ {#eq-discrete-14} Here ${\symbf{X_t}}$ is a row vector of length $k$ of individual-specific covariates, and $\beta^l$ is a $k$-vector of alternative-specific parameters. For identification one category is taken as the base with $\beta^0=0$ (so its numerator is $1$). **Conditional logit.** Here it is the regressors that vary by alternative (e.g. the price or characteristics of choice $j$ itself), while the coefficient vector is common across alternatives: $$ \mbox{Pr}(y_t=l)=\frac{\exp({{\symbf{W}}}_{tl} \beta)}{\sum_{j=0}^J\exp({{\symbf{W}}}_{tj} \beta)} \quad \mbox{for} \ l=0, \dots, J, $$ {#eq-discrete-15} Here ${{\symbf{W}}}_{tj}$ is a row vector of length $k$ of alternative-$j$-specific variables, and $\beta$ is a single $k$-vector of parameters shared across all alternatives $j=0,\dots,J$. For binary and multinomial models alike, the coefficients $\beta$ do not directly measure the effect of a unit change in $X$ on the probability. That effect depends on where in the distribution the change occurs. The average marginal effect (AME) --- averaging the partial effect over the observed distribution of covariates --- is the standard summary. The full treatment of AME vs.\ marginal effects at the mean, including fixed-effects models, is in the [Blog Book](https://xiangao.github.io/blog_book/marginal-effects-fe.html). Writing *both* choice-varying regressors ${{\symbf{W}}}_{tl}$ *and* choice-specific coefficients $\beta^l$ into the same formula, as some treatments do, conflates the two models. It is worth being precise about what that costs, because the usual complaint -- that such a specification is not identified -- is not true. Choice probabilities depend only on utility *differences*, so $\Pr(y_t=l)$ is governed by ${{\symbf{W}}}_{tj}\beta^j - {{\symbf{W}}}_{tl}\beta^l$; when the ${{\symbf{W}}}_{tj}$ vary independently across alternatives, those differences pin down each $\beta^j$ separately, with no normalization required. It is a perfectly estimable model -- it is what Stata's `asclogit` fits when alternative-specific coefficients are requested. What the ambiguous formula really costs is clarity about *which normalization is in force*. Individual-specific regressors require one alternative's coefficients to be fixed ($\beta^0=0$), since only differences across alternatives are identified; alternative-specific regressors entering with a common coefficient require no such restriction. One caveat on "no normalization required" above: it concerns the slope coefficients $\beta^j$. If the specification also carries alternative-specific intercepts, those are identified only up to a location shift and one of them still has to be fixed. A formula that does not say which of the two it means leaves the reader unable to tell what has been normalized, and therefore unable to interpret the coefficients. ### Independence of Irrelevant Alternatives Both logit forms carry a restriction that comes from the softmax itself. Take the ratio of two choice probabilities. With individual-specific regressors, $$ \frac{\Pr(y_t = j)}{\Pr(y_t = l)} = \exp \bigl( {{\symbf{X}}}_t (\beta^j - \beta^l) \bigr), $$ {#eq-discrete-iia} and with alternative-specific regressors and a common coefficient the ratio is $\exp \bigl( ({{\symbf{W}}}_{tj} - {{\symbf{W}}}_{tl}) \beta \bigr)$. Either way everything to do with the other alternatives cancels. The relative odds of $j$ against $l$ do not depend on what else is on the menu, or on the attributes of any third alternative. That is **independence of irrelevant alternatives**, or IIA. Sometimes it is harmless. Often it is not. The standard illustration is travel mode. Suppose commuters split evenly between a car and a red bus, so the odds are 1:1 and each share is $1/2$. Now add a blue bus, identical to the red one in everything but colour. IIA holds the car-to-red-bus odds at 1:1, and the two buses are interchangeable, so the model predicts three shares of $1/3$. What we would expect is that the blue bus draws almost entirely from the red bus, leaving the car near $1/2$. The model has no way to know that the two buses are close substitutes, because the softmax gives it no way to express that. There are two standard escapes. **Nested logit** groups alternatives into nests -- the two buses in one, the car in another -- and allows correlation within a nest, so IIA holds inside nests but not across them. **Mixed logit**, also called random-coefficients logit, lets $\beta$ vary across individuals; shared taste parameters then induce correlation across alternatives and IIA goes away, at the cost of simulation-based estimation. Which one to reach for depends on whether the substitution pattern you are worried about can be described by a grouping.