Welch’s two-sample t-test

The goal of Welch’s two-sample $t$-test to check if the means two unknown population $X \sim \mathcal{N}(\mu_X, \sigma_X)$ and $Y \sim \mathcal{N}(\mu_Y, \sigma_Y)$ are the same or different.

For statistical design considerations related to two-sample tests, see the notebook statistical_design.ipynb.

$\def\stderr#1{\mathbf{se}_{#1}}$ $\def\stderrhat#1{\hat{\mathbf{se}}_{#1}}$ $\def\Mean{\textbf{mean}}$ $\def\Var{\textbf{var}}$ $\def\Std{\textbf{std}}$ $\def\Freq{\textbf{freq}}$ $\def\RelFreq{\textbf{relfreq}}$ $\def\CI#1{\textbf{CI}_{#1}}$ $\def\CIL#1{\textbf{L}_{#1}}$ $\def\CIU#1{\textbf{U}_{#1}}$ $\def\ci#1{\textbf{ci}_{#1}}$ $\def\cil#1{\textbf{l}_{#1}}$ $\def\ciu#1{\textbf{u}_{#1}}$ $\def\DMeans{\textbf{dmeans}}$ $\def\Prop{\textbf{prop}}$ $\def\DProps{\textbf{dprops}}$ $\def\Err{ {\Large \varepsilon}}$ $\def\calN{\mathcal{N}}$ $\def\Tdist{\mathcal{T}}$ $\def\Expon{\mathrm{Expon}}$

(this cell contains the macro definitions like $\stderr{\overline{\mathbf{x}}}$, $\stderrhat{}$, $\Mean$, ...)

Data¶

Two samples of numerical observations $\mathbf{x}=[x_1, x_2, \ldots, x_n]$ and $\mathbf{y}=[y_1, y_2,\ldots, y_m]$ from independent populations.

Assumptions¶

We assume the unknown populations are normally distributed $\textbf{(NORM)}$, or the sample is large enough $\textbf{(LARGEn)}$, so that the sampling distributions of the means in the two populations will be approximately normally distributed:

where $\sigma_X$ and $\sigma_Y$ are the standard deviations of the unknown populations.

Hypotheses¶

$H_0: \mu_X = \mu_Y$ and $H_A: \mu_X \neq \mu_Y$.

Estimates¶

Compute the sample means $\overline{\mathbf{x}} = \Mean(\mathbf{x}) = \frac{1}{n}\sum_{i=1}^n x_i$, $\overline{\mathbf{y}} = \Mean(\mathbf{y}) = \frac{1}{m}\sum_{i=1}^m y_i$, and the observed difference between means $\hat{d} = \DMeans(\mathbf{x}, \mathbf{y}) = \overline{\mathbf{x}} - \overline{\mathbf{y}}$.

We need to also compute the standard deviations $s_{\mathbf{x}} = \Std(\mathbf{x}) = \sqrt{ \frac{1}{n-1}\sum_{i=1}^n (x_i-\overline{\mathbf{x}})^2 }$ and $s_{\mathbf{y}} = \Std(\mathbf{y}) = \sqrt{ \frac{1}{m-1}\sum_{i=1}^m (y_i-\overline{\mathbf{y}})^2 }$, which are estimates for the unknown population standard deviations $\sigma_X$ and $\sigma_Y$.

Formulas¶

The estimated standard error of the difference-between-means estimator is

where $s_{\mathbf{x}}$ and $s_{\mathbf{y}}$ are the sample standard deviations.

The Welch--Satterthwaite formula for the degrees of freedom parameter $\nu_d$ is

In Python, we can calculate $\nu_d$ by calling the helper function calcdf: $\nu_d = \tt{calcdf}(s_{\mathbf{x}}, n, s_{\mathbf{y}}, m)$.

Test statistic¶

Compute the $t$-statistic $t = \frac{\hat{d} - 0}{ \stderrhat{\hat{d}} }$.

Sampling distribution¶

Student’s $t$-distribution with $\nu_d$ degrees of freedom, where the degrees of freedom parameter is computed using the helper function calcdf: $\nu_d = \tt{calcdf}(s_{\mathbf{x}}, n, s_{\mathbf{y}}, m)$, which implements the Welch--Satterthwaite formula.

P-value calculation¶

To obtain the $p$-value, we first compute the observed $t$-statistic, then calculate the tail probabilities in the two tails of the standard normal distribution $T_0 \sim \mathcal{T}(\nu_d)$.

Let’s look at the source code of the function ttest_dmeans, which performs the steps of the two-sample t-test.

import numpy as np
from scipy.stats import t as tdist
from ministats import mean, std, calcdf, tailprobs

def ttest_dmeans(xsample, ysample, alt="two-sided"):
    """
    T-test to detect difference between two populations means
    based on the difference between sample means.
    """
    # Calculate the observed difference between means
    obsdhat = mean(xsample) - mean(ysample)

    # Calculate the sample sizes and the stds
    n, m = len(xsample), len(ysample)
    sx, sy = std(xsample), std(ysample)

    # Calculate the standard error, the degrees of
    # freedom, the null model, and the t-statistic
    seD = np.sqrt(sx**2/n + sy**2/m)
    obst = (obsdhat - 0) / seD
    dfD = calcdf(sx, n, sy, m)
    rvT0 = tdist(df=dfD)

    # Calculate the p-value from the t-distribution
    pvalue = tailprobs(rvT0, obst, alt=alt)
    return pvalue

See Section 3.5 in the No Bullshit Guide to Statistics for the detailed explanation of these steps. You can also check out the section Analytical approximation methods in the notebook notebooks/35_comparing_two_populations.ipynb.

To perform the two-sample $t$-test on the samples xs and ys, we call ttest_dmeans(xs, ys) or ttest_ind(xs, ys, equal_var=False), as we’ll show in the Examples section below.

Effect size estimates¶

import numpy as np
from scipy.stats import t as tdist
from ministats import calcdf

def ci_dmeans(xsample, ysample, alpha=0.1):
    """
    Compute confidence interval for the difference between population means.
    """
    stdX, n = np.std(xsample, ddof=1), len(xsample)
    stdY, m = np.std(ysample, ddof=1), len(ysample)
    dhat = np.mean(xsample) - np.mean(ysample)
    seD = np.sqrt(stdX**2/n + stdY**2/m)
    dfD = calcdf(stdX, n, stdY, m)
    t_l = tdist(df=dfD).ppf(alpha/2)
    t_u = tdist(df=dfD).ppf(1-alpha/2)
    return [dhat + t_l*seD, dhat + t_u*seD]

TODO: narrate code

To use the function ...

Related¶

1. The two-sample t-test is an improved version of the two-sample z-test

2. The Welch’s two-sample t-test always a better option than the two-sample t-test with pooled variance

3. An alternative approach for comparing two groups is to use the permutation test (TODO link to nb)

4. Another alternative is to use the linear model (TODO: deep link to the everything is a LM)

Examples¶

Let’s start by generating some samples from the following three populations for use in the examples we present below.

Note populations $B$ and $C$ are identical.

We’ll now generate a samples of size $n=20$ from each of these populaitons and store them in a Pandas data frame called data.

Example 1: populations are different¶

We’ll now apply the Welch’s two-sample t-test to compare the samples from population A and B, which we know are different.

The sample means seem to reflect the difference between population means too...

... but we must consider the difference between means relative to the standard deviations of the two samples.

Here is a picture that shows the data points, the sample means, and the sample standard deviations for the two samples.

Let’s now use the helper function ttest_dmeans to perform the two-sample t-test calculation and obtain the $p$-value.

The $p$-value we obtain is 0.000595, which is below the cutoff value $\alpha=0.05$, so our conclusion is we reject the null hypothesis: the difference between the means of the two unknown populations is statistically significant.

Effect size¶

The confidence interval for the effect size $\Delta = \mu_A - \mu_B$ is

Example 2: two samples from the same population¶

Let’s now analyze the samples from populations B and C that come from the same distribution, which is the situation described by the null hypothesis.

The $p$-value we obtain is 0.40, which is above the cutoff value $\alpha=0.05$ so our conclusion is that we’ve failed to reject the null hypothesis: the means of two samples are not significantly different.

Effect size¶

The confidence interval for the effect size $\Delta = \mu_B - \mu_C$ is

Example 3: Lalonde dataset¶

see original paper https://business.baylor.edu/scott_cunningham/teaching/lalonde-1986.pdf
and these links for more info: https://www.one-tab.com/page/h_npVXMeTp2T7Dm5fAlDqw

Effect size¶

Example 4: analysis of a dataset that contains outliers¶

The dataset outliers2.csv consists of measurements of “soil score” variable using a specialized equipment for calculating soil quality.

The measurement apparatus has an upper limit beyound which it doesn’t function properly, so all observations that exceed this limit are recorded as the value 2500.

The effect of the outliers in group B is to pull the mean up, to a much higher value than where the other data points are.

The outliers pull the mean in group B to a high value, which leads us to conclude there is a statistically significant difference between the two groups. This is because we’re using the $t$-test when the normality assumption is not valid.

See the example notebook Mann-Whitney_U-test.ipynb for an analysis of this data using the nonparametric test for comparing two groups, which is more appropriate for data with outliers.

Discussion¶

Links¶

1. For more details about the Welch’s two-sample t-test, see the notebook notebooks/35_comparing_two_populations.ipynb, which contains a more detailed derivation.

2. See the Section 3.5 exercises notebook: exercises/sec35_comparing_two_groups.ipynb for practice problems.