44 Power analysis with list experiment

Published

April 20, 2024

This chapter follows Ulrich et al. (2012).

44.1 List experiment

The list experiment (also called the item count technique, Miller 1984) elicits the prevalence of a sensitive attribute without asking respondents to reveal their own status. A control group of $n_c$ respondents receives $k$ neutral items and reports the total count of “yes” answers. An experimental group of $n_e$ respondents receives the same $k$ items plus the sensitive item. The expected counts are $\mu_c = \sum_{i=1}^k \pi_i$ and $\mu_e = \sum_{i=1}^k \pi_i + \pi_S$, so the difference in means estimates $\pi_S$: \[ \hat \pi_S = \hat \mu_e - \hat \mu_c . \tag{44.1}\]

44.2 Power analysis

44.2.1 Define power

Suppose $\pi_s$ is the proportion of the population that has the sensitive attribute, or the proportion of the population that would answer the sensitive question with “yes”, such as the proportion of people who have committed a crime. The null hypothesis $H0$ is that $\pi_s=0$, and the alternative hypothesis $H1$ is that $\pi_s>0$.

The expected mean and the sampling variance of $\hat \pi_s$ under the null hypothesis H0 are denoted $\mu_0$ and $\sigma_0^2$, respectively. Likewise, let $\mu_1$ and $\sigma_1^2$ be the mean and the variance under the alternative hypothesis H1 that $\pi_s$ has a specific value that is greater than zero, for instance, $H1 : \pi_s = 0.05$

statistical power P(“H1”|H1) of a test represents the probability of rejecting $H0$ in favor of $H1$ given that $H1$ is true.

On a general level, this probability is computed as, \[ \begin{aligned} P(“H_1”|H_1) &= P(\hat \pi_s > c_{1-\alpha}) \\ &= P(\frac{\hat \pi_s - \mu_1}{\sigma_1} > \frac{c_{1-\alpha} - \mu_1}{\sigma_1})\\ &= P(Z > \frac{c_{1-\alpha} - \mu_1}{\sigma_1}) \\ &= 1 - \Phi( \frac{c_{1-\alpha} - \mu_1}{\sigma_1}) \\ &= \Phi( \frac{\mu_1-c_{1-\alpha} }{\sigma_1}) \\ &= \Phi( \frac{\mu_1-\mu_0 + z_\alpha \sigma_0 }{\sigma_1}) \\ \end{aligned} \tag{44.2}\]

Consider a special case, $\mu_0=0$, $\mu_1=1$, $\sigma_0=\sigma_1=1$, and $z_\alpha=-1.64$, then the power is 0.26. This is the one side power. If we consider two sided power, then the power is .168, since $z_{\alpha/2}=-1.96$.

44.2.2 Power for list experiment

Under H1, the sampling variance of $\hat \pi_s$ is

\[ \sigma_1^2 = \frac{\pi_s (1-\pi_s)}{n_e} + (\frac{1}{n_e} + \frac{1}{n_c}) \sum_{i=1}^k \pi_i (1-\pi_i) \tag{44.3}\] where $\pi_i$ is the probability of answering the neutral question $N_i$ with “yes”.

Under H0, the sampling variance of $\hat \pi_s$ is \[ \sigma_0^2 = (\frac{1}{n_e} + \frac{1}{n_c}) \sum_{i=1}^k \pi_i (1-\pi_i) \tag{44.4}\]

Assuming $n_e=n_c=n$, then the power is

\[ P("H_1"|H_1) = \Phi( \frac{\pi_s + z_\alpha \sqrt{\frac{2}{n} \sum_{i=1}^k \pi_i (1-\pi_i)}}{\sqrt{\frac{\pi_s(1-\pi_s)}{n} + \frac{2}{n} \sum_{i=1}^k \pi_i (1-\pi_i)}}) \tag{44.5}\]

For example, when you have $\pi_s=.1$, $n=500$, $k=4$, $\pi_i=.1$, then the power is

Code

num=.1+qnorm(.05)*sqrt((1/250)*(4*.1*.9))
denom=sqrt((.1*.9/500+(1/250)*4*.1*.9))
pnorm(num/denom)

[1] 0.8247802

With 500 per arm, 4 neutral questions, a sensitive-item prevalence of 0.1 and neutral-item probabilities of 0.1, the power comes out at 0.825.

44.2.3 Sample size for list experiment

Assuming $n_e=n_c$,

\[ a_1 = \sqrt{2 \pi_s (1-\pi_s) + 4 \sum_{i=1}^k \pi_i (1-\pi_i)} \tag{44.6}\]

\[ a_0 = 2 \sqrt{ \sum_{i=1}^k \pi_i (1-\pi_i)} \tag{44.7}\]

Sample size needed for a power of $1-\beta$ is \[ n = (\frac{(z_{1-\beta}) a_1 - z_\alpha a_0}{\mu_1-\mu_0})^2 \tag{44.8}\]

where $\beta$ is type 2 error, $\alpha$ is type 1 error. Here $n$ is the total sample size; each arm receives $n/2$. The factor $\sqrt 2$ inside $a_1=\sqrt{2\pi_s(1-\pi_s)+4\sum\pi_i(1-\pi_i)}$ and $a_0=2\sqrt{\sum\pi_i(1-\pi_i)}$ converts the per-arm variances $\sigma_1^2,\sigma_0^2$ above into a total-sample formula, so this $n$ is twice the per-group size that the power section uses.

Let’s look at an example: suppose the proportion of the population that would answer the sensitive question with “yes” is 0.1. The probability of answering the neutral question $N_i$ with “yes” is 0.1. The type 1 error is 0.05, and the type 2 error is 0.2. Suppose we have 4 neutral questions. What is the sample size needed?

Code

a1=sqrt(2*.1*.9+4*4*.1*.9)
a0=2*sqrt(4*.1*.9)
n=((qnorm(.8)*a1-qnorm(.05)*a0)/.1)^2
n

[1] 927.2228

The required total sample is 927, so about 464 per arm.

Suppose probability of answering the neutral question $N_i$ with “yes” is .3.

Code

a1=sqrt(2*.1*.9+4*4*.3*.7)
a0=2*sqrt(4*.3*.7)
n=((qnorm(.8)*a1-qnorm(.05)*a0)/.1)^2
n

[1] 2114.682

Raising the neutral-item probability from 0.1 to 0.3 pushes the requirement to 2,115 — more than double. The neutral questions are pure noise from the estimand’s point of view, and their variance $\pi_i(1-\pi_i)$ is maximised at 0.5, so choosing neutral items with probabilities near 0 or 1 is what keeps the design efficient.

Suppose the probability of answering the sensitive question with “yes” is .05.

Code

a1=sqrt(2*.05*.95+4*4*.3*.7)
a0=2*sqrt(4*.3*.7)
n=((qnorm(.8)*a1-qnorm(.05)*a0)/.05)^2
n

[1] 8388.512

Halving the sensitive-item prevalence from 0.1 to 0.05 raises the requirement to 8,389, a fourfold increase. The effect enters the denominator of the sample-size formula squared, so rare sensitive behaviours are expensive to measure this way: the list experiment buys honesty at a steep cost in $n$.

--- title: "Power analysis with list experiment" date: "2024-04-20" --- This chapter follows Ulrich et al. (2012). ## List experiment The list experiment (also called the item count technique, Miller 1984) elicits the prevalence of a sensitive attribute without asking respondents to reveal their own status. A control group of $n_c$ respondents receives $k$ neutral items and reports the total count of “yes” answers. An experimental group of $n_e$ respondents receives the same $k$ items plus the sensitive item. The expected counts are $\mu_c = \sum_{i=1}^k \pi_i$ and $\mu_e = \sum_{i=1}^k \pi_i + \pi_S$, so the difference in means estimates $\pi_S$: $$ \hat \pi_S = \hat \mu_e - \hat \mu_c .$$ {#eq-power-list-experiment-1} ## Power analysis ### Define power Suppose $\pi_s$ is the proportion of the population that has the sensitive attribute, or the proportion of the population that would answer the sensitive question with “yes”, such as the proportion of people who have committed a crime. The null hypothesis $H0$ is that $\pi_s=0$, and the alternative hypothesis $H1$ is that $\pi_s>0$. The expected mean and the sampling variance of $\hat \pi_s$ under the null hypothesis H0 are denoted $\mu_0$ and $\sigma_0^2$, respectively. Likewise, let $\mu_1$ and $\sigma_1^2$ be the mean and the variance under the alternative hypothesis H1 that $\pi_s$ has a specific value that is greater than zero, for instance, $H1 : \pi_s = 0.05$ statistical power P(“H1”|H1) of a test represents the probability of rejecting $H0$ in favor of $H1$ given that $H1$ is true. On a general level, this probability is computed as, $$ \begin{aligned} P(“H_1”|H_1) &= P(\hat \pi_s > c_{1-\alpha}) \\ &= P(\frac{\hat \pi_s - \mu_1}{\sigma_1} > \frac{c_{1-\alpha} - \mu_1}{\sigma_1})\\ &= P(Z > \frac{c_{1-\alpha} - \mu_1}{\sigma_1}) \\ &= 1 - \Phi( \frac{c_{1-\alpha} - \mu_1}{\sigma_1}) \\ &= \Phi( \frac{\mu_1-c_{1-\alpha} }{\sigma_1}) \\ &= \Phi( \frac{\mu_1-\mu_0 + z_\alpha \sigma_0 }{\sigma_1}) \\ \end{aligned} $$ {#eq-power-list-experiment-2} Consider a special case, $\mu_0=0$, $\mu_1=1$, $\sigma_0=\sigma_1=1$, and $z_\alpha=-1.64$, then the power is 0.26. This is the one side power. If we consider two sided power, then the power is .168, since $z_{\alpha/2}=-1.96$. ### Power for list experiment Under H1, the sampling variance of $\hat \pi_s$ is $$ \sigma_1^2 = \frac{\pi_s (1-\pi_s)}{n_e} + (\frac{1}{n_e} + \frac{1}{n_c}) \sum_{i=1}^k \pi_i (1-\pi_i) $$ {#eq-power-list-experiment-3} where $\pi_i$ is the probability of answering the neutral question $N_i$ with “yes”. Under H0, the sampling variance of $\hat \pi_s$ is $$ \sigma_0^2 = (\frac{1}{n_e} + \frac{1}{n_c}) \sum_{i=1}^k \pi_i (1-\pi_i) $$ {#eq-power-list-experiment-4} Assuming $n_e=n_c=n$, then the power is $$ P("H_1"|H_1) = \Phi( \frac{\pi_s + z_\alpha \sqrt{\frac{2}{n} \sum_{i=1}^k \pi_i (1-\pi_i)}}{\sqrt{\frac{\pi_s(1-\pi_s)}{n} + \frac{2}{n} \sum_{i=1}^k \pi_i (1-\pi_i)}}) $$ {#eq-power-list-experiment-5} For example, when you have $\pi_s=.1$, $n=500$, $k=4$, $\pi_i=.1$, then the power is ```{r} num=.1+qnorm(.05)*sqrt((1/250)*(4*.1*.9)) denom=sqrt((.1*.9/500+(1/250)*4*.1*.9)) pnorm(num/denom) ``` With 500 per arm, 4 neutral questions, a sensitive-item prevalence of 0.1 and neutral-item probabilities of 0.1, the power comes out at 0.825. ### Sample size for list experiment Assuming $n_e=n_c$, $$ a_1 = \sqrt{2 \pi_s (1-\pi_s) + 4 \sum_{i=1}^k \pi_i (1-\pi_i)} $$ {#eq-power-list-experiment-6} $$ a_0 = 2 \sqrt{ \sum_{i=1}^k \pi_i (1-\pi_i)} $$ {#eq-power-list-experiment-7} Sample size needed for a power of $1-\beta$ is $$ n = (\frac{(z_{1-\beta}) a_1 - z_\alpha a_0}{\mu_1-\mu_0})^2$$ {#eq-power-list-experiment-8} where $\beta$ is type 2 error, $\alpha$ is type 1 error. Here $n$ is the **total** sample size; each arm receives $n/2$. The factor $\sqrt 2$ inside $a_1=\sqrt{2\pi_s(1-\pi_s)+4\sum\pi_i(1-\pi_i)}$ and $a_0=2\sqrt{\sum\pi_i(1-\pi_i)}$ converts the per-arm variances $\sigma_1^2,\sigma_0^2$ above into a total-sample formula, so this $n$ is twice the per-group size that the power section uses. Let's look at an example: suppose the proportion of the population that would answer the sensitive question with “yes” is 0.1. The probability of answering the neutral question $N_i$ with “yes” is 0.1. The type 1 error is 0.05, and the type 2 error is 0.2. Suppose we have 4 neutral questions. What is the sample size needed? ```{r} a1=sqrt(2*.1*.9+4*4*.1*.9) a0=2*sqrt(4*.1*.9) n=((qnorm(.8)*a1-qnorm(.05)*a0)/.1)^2 n ``` The required total sample is 927, so about 464 per arm. Suppose probability of answering the neutral question $N_i$ with “yes” is .3. ```{r} a1=sqrt(2*.1*.9+4*4*.3*.7) a0=2*sqrt(4*.3*.7) n=((qnorm(.8)*a1-qnorm(.05)*a0)/.1)^2 n ``` Raising the neutral-item probability from 0.1 to 0.3 pushes the requirement to 2,115 — more than double. The neutral questions are pure noise from the estimand's point of view, and their variance $\pi_i(1-\pi_i)$ is maximised at 0.5, so choosing neutral items with probabilities near 0 or 1 is what keeps the design efficient. Suppose the probability of answering the sensitive question with “yes” is .05. ```{r} a1=sqrt(2*.05*.95+4*4*.3*.7) a0=2*sqrt(4*.3*.7) n=((qnorm(.8)*a1-qnorm(.05)*a0)/.05)^2 n ``` Halving the sensitive-item prevalence from 0.1 to 0.05 raises the requirement to 8,389, a fourfold increase. The effect enters the denominator of the sample-size formula squared, so rare sensitive behaviours are expensive to measure this way: the list experiment buys honesty at a steep cost in $n$.