48 Equivalence testing and two one-sided tests (TOST)

Published

July 24, 2026

48.1 Why non-significance is not evidence of no effect

Suppose we use the usual null hypothesis significance test:

\[H_0: \theta = 0 \quad \text{vs.} \quad H_1: \theta \neq 0\]

Failing to reject $H_0$ ($p > 0.05$) does not show that $\theta = 0$. We can get a large $p$-value because the true effect is small, but we can also get one because the sample is small or the outcome is noisy.

The usual test cannot tell these cases apart. If we want evidence that an effect is small enough to ignore, we need an equivalence test.

48.2 Two one-sided tests

We first choose a smallest effect size of interest. I write the resulting equivalence interval as $[-\Delta, +\Delta]$. An effect inside this interval is small enough to be negligible for the application.

The two one-sided tests (TOST) procedure (Schuirmann, 1987; Lakens, 2017) uses two null hypotheses:

\[ H_{01}: \theta \le -\Delta \quad \text{and} \quad H_{02}: \theta \ge +\Delta \]

If we reject both null hypotheses at level $\alpha$, we conclude that $-\Delta < \theta < +\Delta$ with $(1 - 2\alpha)$ confidence.

An equivalent way to do the test is to check whether the $(1 - 2\alpha)$ confidence interval falls entirely inside $[-\Delta, +\Delta]$. When $\alpha = 0.05$, we use the 90% confidence interval.

48.3 Using `TOSTER` in R

Let’s use the 1974 Motor Trend data in mtcars. We compare quarter-mile acceleration time (qsec) between automatic ($n = 19$) and manual ($n = 13$) transmissions.

Code

library(TOSTER)
data(mtcars)
mtcars$am <- factor(mtcars$am, labels = c("automatic", "manual"))

# Standard two-sample t-test
t.test(qsec ~ am, data = mtcars)


    Welch Two Sample t-test

data:  qsec by am
t = 1.2878, df = 25.534, p-value = 0.2093
alternative hypothesis: true difference in means between group automatic and group manual is not equal to 0
95 percent confidence interval:
 -0.4918522  2.1381679
sample estimates:
mean in group automatic    mean in group manual 
               18.18316                17.36000

The usual two-sample $t$-test gives $p = 0.21$. This tells us that the data do not clearly distinguish the groups, but it is not evidence that they are equivalent.

48.3.1 TOST with $\Delta = 1$ second

Suppose we pre-specify that a difference of less than 1 second ($\Delta = 1$) is negligible:

Code

t_TOST(formula = qsec ~ am, data = mtcars, eqb = 1, var.equal = FALSE)


Welch Two Sample t-test

The equivalence test was non-significant, t(25.53) = -0.28, p = 0.39
The null hypothesis test was non-significant, t(25.53) = 1.288, p = 0.21
NHST: don't reject null significance hypothesis that the effect is equal to zero 
TOST: don't reject null equivalence hypothesis

TOST Results 
                 t    df p.value
t-test      1.2878 25.53   0.209
TOST Lower  2.8524 25.53   0.004
TOST Upper -0.2767 25.53   0.392

Effect Sizes 
               Estimate     SE              C.I. Conf. Level
Raw              0.8232 0.6392 [-0.2678, 1.9141]         0.9
Hedges's g(av)   0.4522 0.3783 [-0.1375, 1.0342]         0.9
Note: SMD confidence intervals are an approximation. See vignette("SMD_calcs").

The 90% confidence interval is $[-0.27, 1.91]$, so it extends beyond $+1.0$. We fail to reject $H_{02}$ and cannot claim equivalence within $\pm 1$ second.

48.3.2 TOST with $\Delta = 2$ seconds

If the substantive tolerance is instead $\Delta = 2$ seconds:

Code

t_TOST(formula = qsec ~ am, data = mtcars, eqb = 2, var.equal = FALSE)


Welch Two Sample t-test

The equivalence test was significant, t(25.53) = -1.8, p = 0.04
The null hypothesis test was non-significant, t(25.53) = 1.288, p = 0.21
NHST: don't reject null significance hypothesis that the effect is equal to zero 
TOST: reject null equivalence hypothesis

TOST Results 
                t    df p.value
t-test      1.288 25.53   0.209
TOST Lower  4.417 25.53 < 0.001
TOST Upper -1.841 25.53   0.039

Effect Sizes 
               Estimate     SE              C.I. Conf. Level
Raw              0.8232 0.6392 [-0.2678, 1.9141]         0.9
Hedges's g(av)   0.4522 0.3783 [-0.1375, 1.0342]         0.9
Note: SMD confidence intervals are an approximation. See vignette("SMD_calcs").

Now the 90% confidence interval $[-0.27, 1.91]$ lies completely inside $[-2, 2]$. Both one-sided tests reject ($p_1 < 0.001$, $p_2 = 0.039$), so the groups are statistically equivalent within $\pm 2$ seconds at $\alpha = 0.05$.

48.4 Bayesian equivalence testing and units

We can also evaluate a Bayes factor for an interval null:

\[\text{BF}_{\text{in/out}} = \frac{p(\text{data} \mid \theta \in [-\Delta, +\Delta])}{p(\text{data} \mid \theta \notin [-\Delta, +\Delta])}\]

The two R packages use different units. TOSTER::t_TOST(eqb = ...) takes the bound in the original outcome units, which are seconds here. BayesFactor::ttestBF(nullInterval = ...) expects a standardized effect size in Cohen’s $d$.

We therefore need to divide the raw bound by the pooled standard deviation before passing it to ttestBF:

Code

library(BayesFactor)

# Compute pooled SD to standardize the 2-second bound
grp <- split(mtcars$qsec, mtcars$am)
n1 <- length(grp[[1]])
n2 <- length(grp[[2]])
sd_pooled <- sqrt(((n1 - 1) * var(grp[[1]]) + (n2 - 1) * var(grp[[2]])) / (n1 + n2 - 2))

delta_d <- 2 / sd_pooled  # convert +/- 2 seconds to Cohen's d
c(sd_pooled = sd_pooled, delta_d = delta_d)

sd_pooled   delta_d 
 1.767842  1.131323

With $s_{\text{pooled}} \approx 1.77$, $\Delta = 2$ corresponds to $d \approx 1.13$. We can now test the interval null:

Code

ttestBF(formula = qsec ~ am, data = mtcars, nullInterval = c(-delta_d, delta_d))

Bayes factor analysis
--------------
[1] Alt., r=0.707 -1.1313225176573<d<1.1313225176573    : 0.9813314  ±0%
[2] Alt., r=0.707 !(-1.1313225176573<d<1.1313225176573) : 0.02203644 ±0.03%

Against denominator:
  Null, mu1-mu2 = 0 
---
Bayes factor type: BFindepSample, JZS

The Bayes factor is 0.98 for an effect inside $[-\Delta, +\Delta]$ relative to the point null and 0.022 for an effect outside it. Their ratio is about $45:1$ ($0.98 / 0.022 \approx 44.5$) in favor of the equivalence interval. This agrees with the TOST result.

48.5 Choosing the equivalence bound

The data cannot choose $\Delta$ for us. It has to come from the application. We might use the smallest effect with a meaningful practical or economic consequence, or the smallest effect that would justify the cost of an intervention. Another option is Simonsohn’s (2015) “small telescopes” benchmark: use the effect size that an earlier study had only 33% power to detect, denoted by $d_{33\%}$. The 33% refers to power, not to 33% of the original estimate; $d_{33\%}$ is a little more than half of $d_{80\%}$.

48.6 What to report

A large $p$-value is not evidence of no effect. For TOST, report the point estimate, the pre-specified equivalence bound, the $(1 - 2\alpha)$ confidence interval, and the two test decisions. This gives readers enough information to see whether another reasonable choice of $\Delta$ would change the conclusion. For a Bayesian interval test, also check whether the software expects the bound in the original units or in standardized units.

--- title: "Equivalence testing and two one-sided tests (TOST)" date: "2026-07-24" --- ## Why non-significance is not evidence of no effect Suppose we use the usual null hypothesis significance test: $$H_0: \theta = 0 \quad \text{vs.} \quad H_1: \theta \neq 0$$ Failing to reject $H_0$ ($p > 0.05$) does not show that $\theta = 0$. We can get a large $p$-value because the true effect is small, but we can also get one because the sample is small or the outcome is noisy. The usual test cannot tell these cases apart. If we want evidence that an effect is small enough to ignore, we need an equivalence test. ## Two one-sided tests We first choose a smallest effect size of interest. I write the resulting equivalence interval as $[-\Delta, +\Delta]$. An effect inside this interval is small enough to be negligible for the application. The two one-sided tests (TOST) procedure (Schuirmann, 1987; Lakens, 2017) uses two null hypotheses: $$ H_{01}: \theta \le -\Delta \quad \text{and} \quad H_{02}: \theta \ge +\Delta $$ If we reject both null hypotheses at level $\alpha$, we conclude that $-\Delta < \theta < +\Delta$ with $(1 - 2\alpha)$ confidence. An equivalent way to do the test is to check whether the $(1 - 2\alpha)$ confidence interval falls entirely inside $[-\Delta, +\Delta]$. When $\alpha = 0.05$, we use the 90% confidence interval. ## Using `TOSTER` in R Let's use the 1974 Motor Trend data in `mtcars`. We compare quarter-mile acceleration time (`qsec`) between automatic ($n = 19$) and manual ($n = 13$) transmissions. ```{r} #| message: false library(TOSTER) data(mtcars) mtcars$am <- factor(mtcars$am, labels = c("automatic", "manual")) # Standard two-sample t-test t.test(qsec ~ am, data = mtcars) ``` The usual two-sample $t$-test gives $p = 0.21$. This tells us that the data do not clearly distinguish the groups, but it is not evidence that they are equivalent. ### TOST with $\Delta = 1$ second Suppose we pre-specify that a difference of less than 1 second ($\Delta = 1$) is negligible: ```{r} t_TOST(formula = qsec ~ am, data = mtcars, eqb = 1, var.equal = FALSE) ``` The 90% confidence interval is $[-0.27, 1.91]$, so it extends beyond $+1.0$. We fail to reject $H_{02}$ and cannot claim equivalence within $\pm 1$ second. ### TOST with $\Delta = 2$ seconds If the substantive tolerance is instead $\Delta = 2$ seconds: ```{r} t_TOST(formula = qsec ~ am, data = mtcars, eqb = 2, var.equal = FALSE) ``` Now the 90% confidence interval $[-0.27, 1.91]$ lies completely inside $[-2, 2]$. Both one-sided tests reject ($p_1 < 0.001$, $p_2 = 0.039$), so the groups are statistically equivalent within $\pm 2$ seconds at $\alpha = 0.05$. ## Bayesian equivalence testing and units We can also evaluate a Bayes factor for an interval null: $$\text{BF}_{\text{in/out}} = \frac{p(\text{data} \mid \theta \in [-\Delta, +\Delta])}{p(\text{data} \mid \theta \notin [-\Delta, +\Delta])}$$ The two R packages use different units. `TOSTER::t_TOST(eqb = ...)` takes the bound in the original outcome units, which are seconds here. `BayesFactor::ttestBF(nullInterval = ...)` expects a standardized effect size in Cohen's $d$. We therefore need to divide the raw bound by the pooled standard deviation before passing it to `ttestBF`: ```{r} #| message: false library(BayesFactor) # Compute pooled SD to standardize the 2-second bound grp <- split(mtcars$qsec, mtcars$am) n1 <- length(grp[[1]]) n2 <- length(grp[[2]]) sd_pooled <- sqrt(((n1 - 1) * var(grp[[1]]) + (n2 - 1) * var(grp[[2]])) / (n1 + n2 - 2)) delta_d <- 2 / sd_pooled # convert +/- 2 seconds to Cohen's d c(sd_pooled = sd_pooled, delta_d = delta_d) ``` With $s_{\text{pooled}} \approx 1.77$, $\Delta = 2$ corresponds to $d \approx 1.13$. We can now test the interval null: ```{r} ttestBF(formula = qsec ~ am, data = mtcars, nullInterval = c(-delta_d, delta_d)) ``` The Bayes factor is 0.98 for an effect inside $[-\Delta, +\Delta]$ relative to the point null and 0.022 for an effect outside it. Their ratio is about $45:1$ ($0.98 / 0.022 \approx 44.5$) in favor of the equivalence interval. This agrees with the TOST result. ## Choosing the equivalence bound The data cannot choose $\Delta$ for us. It has to come from the application. We might use the smallest effect with a meaningful practical or economic consequence, or the smallest effect that would justify the cost of an intervention. Another option is Simonsohn's (2015) "small telescopes" benchmark: use the effect size that an earlier study had only 33% power to detect, denoted by $d_{33\%}$. The 33% refers to power, not to 33% of the original estimate; $d_{33\%}$ is a little more than half of $d_{80\%}$. ## What to report A large $p$-value is not evidence of no effect. For TOST, report the point estimate, the pre-specified equivalence bound, the $(1 - 2\alpha)$ confidence interval, and the two test decisions. This gives readers enough information to see whether another reasonable choice of $\Delta$ would change the conclusion. For a Bayesian interval test, also check whether the software expects the bound in the original units or in standardized units.

48 Equivalence testing and two one-sided tests (TOST)

48.1 Why non-significance is not evidence of no effect

48.2 Two one-sided tests

48.3 Using `TOSTER` in R

48.3.1 TOST with \(\Delta = 1\) second

48.3.2 TOST with \(\Delta = 2\) seconds

48.4 Bayesian equivalence testing and units

48.5 Choosing the equivalence bound

48.6 What to report

48.1 Why non-significance is not evidence of no effect

48.2 Two one-sided tests

48.3 Using TOSTER in R

48.3.1 TOST with \(\Delta = 1\) second

48.3.2 TOST with \(\Delta = 2\) seconds

48.4 Bayesian equivalence testing and units

48.5 Choosing the equivalence bound

48.6 What to report

48.3 Using `TOSTER` in R