Probability basics: axioms, independence, total probability and Bayes

Sample spaces, events and the axioms

• The sample space $S$ is the set of all outcomes. An event is a subset of $S$.
• Rolling two dice gives $\lvert S\rvert = 36$ equally likely ordered pairs.

Kolmogorov's axioms.

1. $P(A) \ge 0$.
2. $P(S) = 1$.
3. For mutually exclusive (disjoint) events, $P(A \cup B \cup \dots) = P(A) + P(B) + \dots$

Consequences.

• $P(A^c) = 1 - P(A)$ and $P(\varnothing) = 0$.
• If $A \subseteq B$, then $P(A) \le P(B)$.
• $P(A \cup B) = P(A) + P(B) - P(A \cap B)$.
• $P(A \cup B \cup C) = \sum P(A) - \sum P(A \cap B) + P(A \cap B \cap C)$.
• Union bound: $P(A \cup B) \le P(A) + P(B)$.

Example 1. $P(A) = 0.5$, $P(B) = 0.4$ and $P(A \cap B) = 0.2$.

• $P(A \cup B) = 0.7$.
• $P(\text{neither}) = 0.3$.
• $P(\text{exactly one}) = 0.7 - 0.2 = 0.5$.
• $P(A \cap B^c) = 0.5 - 0.2 = 0.3$.

Example 2: two dice.

• $P(\text{sum} = 7) = \frac{6}{36} = \frac16$.
• $P(\text{at least one six}) = 1 - \left(\frac56\right)^2 = \frac{11}{36}$.
• $P(\text{doubles}) = \frac16$.

Example 3: the complement trick. The probability of at least one six in 4 rolls is $1 - (5/6)^4 \approx 0.518$. "At least one" questions are almost always easier through the complement.

Conditional probability

$P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0.$

Conditioning shrinks the sample space to $B$. Multiplication rule: $P(A \cap B) = P(B)\,P(A \mid B) = P(A)\,P(B \mid A)$. For more events, $P(A \cap B \cap C) = P(A)P(B \mid A)P(C \mid A \cap B)$.

Example 4. Two cards are drawn without replacement. $P(\text{both red}) = \frac{26}{52}\cdot\frac{25}{51} = \frac{25}{102} \approx 0.245$.

Example 5: conditioning on partial information. A family has two children, and at least one is a boy. What is the probability that both are boys?

• The equally likely cases are BB, BG and GB (GG is ruled out).
• So the answer is $\frac13$, not $\frac12$.

If instead we are told the elder child is a boy, the cases are BB and BG, so the answer is $\frac12$. The exact information matters.