Statistical inference: CLT, confidence intervals and tests

From samples to populations

Vocabulary.

• A population is described by parameters, such as the mean $\mu$, the variance $\sigma^2$ or a proportion $p$. Usually they are unknown.
• A sample $X_1, \dots, X_n$ is drawn from it, independently and from the same distribution (i.i.d.).
• A statistic is any function of the sample. Its probability distribution over repeated samples is its sampling distribution.

The sample mean $\bar X = \frac1n\sum X_i$ has

$E[\bar X] = \mu, \qquad \operatorname{Var}(\bar X) = \frac{\sigma^2}{n}, \qquad \text{SE}(\bar X) = \frac{\sigma}{\sqrt n}.$

The standard error shrinks like $1/\sqrt n$: four times the data halves the error.

The sample variance $S^2 = \frac{1}{n - 1}\sum (X_i - \bar X)^2$ is unbiased: $E[S^2] = \sigma^2$. Dividing by $n$ would underestimate $\sigma^2$ on average, because the deviations are measured from $\bar X$, which sits closer to the data than $\mu$ does. This adjustment is Bessel's correction, and $n - 1$ is the number of degrees of freedom.

Example 1. For the sample 2, 4, 6, 8: $\bar x = 5$, and the squared deviations are 9, 1, 1, 9, which sum to 20.

• The sample variance is $s^2 = 20/3 \approx 6.67$, so $s \approx 2.58$.
• Dividing by $n$ instead would give 5.

Two limit theorems

Law of large numbers. $\bar X_n \to \mu$ as $n \to \infty$. Long-run averages settle down.

Central limit theorem (CLT). If the $X_i$ are i.i.d. with mean $\mu$ and finite variance $\sigma^2$, then for large $n$

$\bar X_n \approx N\!\left(\mu, \frac{\sigma^2}{n}\right), \qquad \text{that is,} \qquad \frac{\bar X_n - \mu}{\sigma/\sqrt n} \approx N(0, 1),$

whatever the shape of the original distribution. Likewise, the sum satisfies $\sum X_i \approx N(n\mu, n\sigma^2)$. A common rule of thumb is that $n \ge 30$ is enough, unless the distribution is very skewed.

Standard normal values to know.

$z$	1	1.28	1.645	1.96	2	2.326	2.576	3
$\Phi(z)$	0.8413	0.90	0.95	0.975	0.9772	0.99	0.995	0.9987

About 68%, 95% and 99.7% of a normal distribution lies within 1, 2 and 3 standard deviations of the mean.

Solved examples

Example 2. Exam scores have mean 50 and standard deviation 10, with an unknown shape. For a random sample of 100 scores, find $P(\bar X > 52)$.

• $\text{SE} = 10/\sqrt{100} = 1$, so $z = (52 - 50)/1 = 2$.
• $P \approx 1 - 0.9772 = 0.0228$.

Example 3: a sum. The service times of 36 customers are independent, each with mean 4 minutes and standard deviation 3 minutes. Find the probability that the total exceeds 160 minutes.

• The total has mean $144$ and standard deviation $3\sqrt{36} = 18$.
• $z = 16/18 \approx 0.89$, so $P \approx 1 - 0.813 = 0.187$.

Example 4: the normal approximation to the binomial. If $X \sim \text{Bin}(n, p)$ with $np$ and $n(1 - p)$ both at least about 10, then $X \approx N(np, np(1 - p))$. Use the continuity correction of $\pm 0.5$.

For $n = 100$ fair coin tosses, find $P(X \ge 60)$.

• The mean is 50 and the standard deviation is 5.
• $P(X \ge 60) \approx P\left(Z \ge \frac{59.5 - 50}{5}\right) = P(Z \ge 1.9) = 0.0287$.

Point estimation

An estimator $\hat\theta$ is a statistic used to guess the parameter $\theta$.

• Bias $= E[\hat\theta] - \theta$. An estimator is unbiased if the bias is 0.
• Mean squared error: $\text{MSE} = E[(\hat\theta - \theta)^2] = \operatorname{Var}(\hat\theta) + \text{bias}^2$.
• An estimator is consistent if $\hat\theta \to \theta$ as $n \to \infty$.

Methods.

• Method of moments: set the sample moments equal to the population moments.
• Maximum likelihood: maximise $\prod f(x_i; \theta)$.

Example 5. For $X \sim \text{Uniform}(0, \theta)$, $E[X] = \theta/2$.

• The method-of-moments estimate is $\hat\theta = 2\bar X$.
• The MLE is $\max_i X_i$: the likelihood $\theta^{-n}$ falls as $\theta$ grows, but $\theta$ must be at least every observation.
• For the data 1, 4, 3: the method of moments gives $2 \times \frac83 \approx 5.33$, and the MLE gives 4.

Example 6: biased can be better. For normal data, $\frac1n\sum(X_i - \bar X)^2$ is biased, yet it has a smaller MSE than $S^2$. The MSE trades bias against variance, just as in machine learning.

Confidence intervals

Idea. A 95% confidence interval is a procedure that captures the true parameter in 95% of repeated samples. For one computed interval, the parameter is either inside it or not. "95%" describes the method, not this particular interval.

Mean, with $\sigma$ known (the z-interval)

$\bar x \pm z_{\alpha/2}\,\frac{\sigma}{\sqrt n}, \qquad z_{0.025} = 1.96 \text{ for } 95\%, \quad z_{0.005} = 2.576 \text{ for } 99\%.$