Literature

Today’s lecture

Based on: Ch. 2-6 in Wackerly, Mendenhall, and Scheaffer (2014)

Concepts of probability

Cox (1946), Berger (2013)

Monty Hall problem

Suppose you’re on a game show, and you’re given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what’s behind the doors, opens another door, say No. 3, which has a goat. He then says to you, “Do you want to pick door No. 2?” Is it to your advantage to switch your choice?

Birthday paradox

23 people stand in a room. What is the probability that 2 people have the same birthday (i.e., only day and month)?

Rolling the dice

Probability of rolling seven 1’s out of 10 dice?

Are the dice balanced?

More difficult questions

What is the probability that the Big Bang gave rise to at least one life-harboring planet in the Universe?

What is the probability for a physical constant to lie within certain limits

Probability views

Frequentist view

Relative frequency over infinitely many repetitions (i.e., ensemble)¹

$$P(\text{event})=\underset{N\to \mathrm{\infty }}{lim}\frac{\mathrm{\#}\text{ events}}{N}$$

Subjective view

Reasonable expectation and personal belief,² e.g., Bayesian view

Probability space

Probability space

triple $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$

$\mathrm{\Omega }$

Sample space: set of possible outcomes from an experiment

$\mathcal{F}$

Event space: set of all possible subsets of $\mathrm{\Omega }$

$\mathbb{P}$

Probability measure: mapping from event $E\subset \mathrm{\Omega }$ to number in $[0,1]$

Sample space

Sample space

set/space of all elementary outcomes ${\omega }_k\in \mathrm{\Omega }$ of a probability experiment, elementary outcomes cannot be further dissected

Discrete sample space

$\mathrm{\Omega }=\{{\omega }_1,\dots ,{\omega }_K\}$, (e.g., $\mathrm{\Omega }=\mathbb{N}$); finite or countable

Continuous sample space

e.g., $\mathrm{\Omega }=\mathbb{R}$, $\mathrm{\Omega }=[0,1{]}^d$

Events

Events E

set of subsets of sample space—must form a $\sigma$-algebra¹

Roll the dice: $\mathrm{\Omega }=\{1,2,3,4,5,6\}$

• $E_0=\mathrm{\varnothing }$
• $E_1=\{2,4,6\}$ (“even numbers”)
• $E_2=\{1,3,5\}$ (“odd numbers”)
• $E_3=\mathrm{\Omega }$
• …

Partition¹

Set $\{A_1,\dots ,A_K\}$ such that

1. $A_i\cap A_j=0\mathrm{\forall }i\ne j$
2. $\mathrm{\Omega }=A_1\cup A_2\cup \cdots \cup A_K$

then for any $B\in \mathrm{\Omega }:B=(B\cap A_1)\cup (B\cap A_2)\cup \cdots \cup (B\cap A_K)$

Probability measure

Function $P:E\to [0,1]$ with the following axioms/properties for any $A\in E$:

1. $P(A)\ge 0$
2. $P(\mathrm{\Omega })=1$
3. Sum rule: $A_i\cap A_j=\mathrm{\varnothing }\mathrm{\forall }i\ne j\Rightarrow P(A_1\cup A_2\cup \dots )=\sum _{i=1}^{\mathrm{\infty }}P(A_i)$

Probability measure

Probability function, $P$, is not given for any sample space, it needs to be contructed for any probability experiment. e.g.,

1. Prob. function for throwing a single fairly-balanced dice: uniform
2. Sample space for throwing two dice $\mathrm{\Omega }=\{(m,n)|m,n\in \{1,2,3,4,5,6\}\}$. Prob. of getting both numbers above 4?
3. A and B play two tennis games. Odds are 2:1 for A to win any game. What is the probability that A wins at least one game? (example 2.4 in Wackerly, Mendenhall, and Scheaffer (2014))

Intersections, unions, subsets

Basic set rules: DeMorgan’s law

Basic set rules (cont’d)

Combinatorial rules

23 people stand in a room. What is the probability that 2 people have the same birthday (i.e., only day and month)?

Counting permutations¹

Number of possibilities to assign $N$ distinct objects to $K\le N$ slots

• Without replacement $$N\times (N-1)\times \cdots \times (N-K+1)=\frac{N!}{(N-K)!}$$

• With replacement $$N^K$$

Combinatorial rules

23 people stand in a room. What is the probability that 2 people have the same birthday (i.e., only day and month)?

from math import factorial

num_people = 23
num_ele_space = 365**num_people
num_ele_distinct = factorial(365) / factorial(365-num_people)
prob = 1. - num_ele_distinct / num_ele_space

print(f'Probability for same birthday is {prob:5.3f}')

Probability for same birthday is 0.507

Conditional probability

Conditional probability of event A given that event B has occurred $$P(A|B)=\frac{P(A\cap B)}{P(B)}$$ If A and B are independent, then $$P(A\cap B)=P(A)P(B)$$

Additive law

Law of total probability¹

Assume $\{A_1,A_2,\dots ,A_K\}$ is a partition of $\mathrm{\Omega }$, then $$P(B)=\sum _{i=1}^{K}P(B|A_i)P(A_i)=\sum _{i=1}^{K}P(B,A_i)$$

$P(B,A_i)$

the joint (or bivariate) probability function for $B$ and $A_i$

$P(B)$

is called a marginal probability

Bayes’ rule¹

$$P(A_j|B)=\frac{P(B|A_j)P(A_j)}{\sum _{i=1}^{K}P(B|A_i)P(A_i)}=\frac{P(B|A_j)P(A_j)}{P(B)}$$

$P(A_j)$

prior (or a-priori) probability

$P(A_j|B)$

posterior probability

$P(B|A_j)$

likelihood

Summary

• Many scientific problems are probabilistic
• Concept/philosophy: frequentist vs. subjective (Bayesian)
• Mathematical framework: Probability space triplet $(\mathrm{\Omega },\mathcal{F},\mathbb{P})$: sample space, event space, probability measure
• Operations on sets: Union, intersection, complement
• Probability laws: conditional probability, additive law, law of total probability, Bayes’ rule