Chapter 01 - Probability Models and Conditioning

Events and outcomes

Theory

Theory 1

Events and outcomes – informally

• An event is a description of something that can happen.
• An outcome is a complete description of something that can happen.

All outcomes are events. An event is usually a partial description. Outcomes are events given with a complete description.

Here ‘complete’ and ‘partial’ are within the context of the probability model.

It can be misleading to say that an ‘outcome’ is an ‘observation’.

• ‘Observations’ occur in the real world, while ‘outcomes’ occur in the model.
• To the extent the model is a good one, and the observation conveys complete information, we can say ‘outcome’ for the observation.

Notice: Because outcomes are complete, no two distinct outcomes could actually happen in a run of the experiment being modeled.

When an event happens, the fact that it has happened constitutes information.

Events and outcomes – mathematically

• The sample space is the set of possible outcomes, so it is the set of the complete descriptions of everything that can happen.
• An event is a subset of the sample space, so it is a collection of outcomes.

For mathematicians: some “wild” subsets are not valid events. Problems with infinity and the continuum…

Notation

• Write $S$ for the set of possible outcomes, $s\in S$ for a single outcome in $S$.

• Write $A,B,C,\cdots \subset S$ or $A_1,A_2,A_3,\cdots \subset S$ for some events, subsets of $S$.

• Write $\mathcal{F}$ for the collection of all events. This is frequently a huge set!

• Write $|A|$ for the cardinality or size of a set $A$, i.e. the number of elements it contains.

Using this notation, we can consider an outcome itself as an event by considering the “singleton” subset $\{\omega \}\subset S$ which contains that outcome alone.

Link to original

Illustration

Example - Coin flipping

Coin flipping

Flip a fair coin two times and record both results.

• Outcomes: sequences, like $HH$ or $TH$.

• Sample space: all possible sequences, i.e. the set $S=\{HH,HT,TH,TT\}$.

• Events: for example:

  • $A=\{HH,HT\}=\text{“first was heads”}$
  • $B=\{HT,TH\}=\text{“exactly one heads”}$
  • $C=\{HT,TH,HH\}=\text{“at least one heads”}$

With this setup, we may combine events in various ways to generate other events:

Complex events: for example: $A\cap B=\{HT\}$, or in words:

$$\text{“first was heads”}\text{AND}\text{“exactly one heads”}=\text{“heads-then-tails”}$$

Notice that the last one is a complete description, namely the outcome $HT$.
$A\cup B=\{HH,HT,TH\}$, or in words:

$$\begin{array}{c}\text{“first was heads”}\text{OR}\text{“exactly one heads”}\\ =\text{“starts with heads, else it’s tails-then-heads”}\end{array}$$ Link to original

Practice exercise

Coin flipping: counting subsets

Flip a fair coin five times and record the results.

How many elements are in the sample space? (How big is $S$?)
How many events are there? (How big is $\mathcal{F}$?)

Solution

There are $2^5=32$ possible sequences, so $|S|=32$.

To count the number of possible subsets, consider that we have 32 distinct items, and a subset is uniquely determined by the binary information – for each item – of whether it is in or out. Thus there are $2^{32}$ possibilities. So $|\mathcal{F}|=2^{32}$.

Link to original

Theory

Theory 2

New events from old

Given two events $A$ and $B$, we can form new events using set operations:

$$\begin{array}{rl}A\cup B⟷& \text{“event }A\text{ OR }\text{event }B\text{”}\\ & \\ A\cap B⟷& \text{“event }A\text{ AND }\text{event }B\text{”}\\ & \\ A^c⟷& \text{not}\text{ event }A\end{array}$$

We also use these terms for events $A$ and $B$:

• They are mutually exclusive when $A\cap B=\varnothing$, that is, they have no elements in common.

• They are collectively exhaustive $A\cup B=S$, that is, when they jointly cover all possible outcomes.

In probability texts, sometimes $A\cap B$ is written “$A\cdot B$” or even (frequently!) “$AB$”.

Rules for sets

Algebraic rules

• Associativity: $(A\cup B)\cup C=A\cup (B\cup C)$. Analogous to $(A+B)+C=A+(B+C)$.

• Distributivity: $A\cap (B\cup C)=(A\cap B)\cup (A\cap C)$. Analogous to $A(B+C)=AB+AC$.

De Morgan’s Laws

• $(A\cup B{)}^c=A^c\cap B^c$

• $(A\cap B{)}^c=A^c\cup B^c$
  In other words: you can distribute “ ${}^c$ ” but must simultaneously do a switch $\cap \leftrightarrow \cup$.

Link to original

Probability models

Theory

Theory 1

Axioms of probability

A probability measure is a function $P:\mathcal{F}\to \mathbb{R}$ satisfying:

Kolmogorov Axioms:

• Axiom 1: $P[A]\ge 0$ for every event $A$
  (probabilities are not negative!)

• Axiom 2: $P[S]=1$
  (probability of “anything” happening is 1)

• Axiom 3: additivity for any countable collection of mutually exclusive events:

$$P[A_1\cup A_2\cup A_3\cup \cdots ]=P[A_1]+P[A_2]+P[A_3]+\cdots$$ $$\text{when:}{A}_i\cap A_j=\varnothing \text{for all }i\ne j$$

Notation: we write $P[A]$ instead of $P(A)$, even though $P$ is a function, to emphasize the fact that $A$ is a set.

Probability model

A probability model or probability space consists of a triple $(S,\mathcal{F},P)$:

• $S$ the sample space

• $\mathcal{F}$ the set of valid events, where every $A\in \mathcal{F}$ satisfies $A\subset S$

• $P:\mathcal{F}\to \mathbb{R}$ a probability measure satisfying the Kolmogorov Axioms

Finitely many exclusive events

It is a consequence of the Kolmogorov Axioms that additivity also works for finite collections of mutually exclusive events:

$$\begin{array}{c}P[A\cup B]=P[A]+P[B]\\ \\ P[A_1\cup \cdots \cup A_n]=P[A_1]+\cdots +P[A_n]\end{array}$$

Inferences from Kolmogorov

A probability measure satisfies these rules.
They can be deduced from the Kolmogorov Axioms.

• Negation: Can you find $P[A^c]$ but not $P[A]$? Use negation:

$$P[A]=1-P[A^c]$$

• Monotonicity: Probabilities grow when outcomes are added:

$$A\subset B\gg \gg P[A]\le P[B]$$

• Inclusion-Exclusion: A trick for resolving unions:

$$P[A\cup B]=P[A]+P[B]-P[A\cap B]$$

(even when $A$ and $B$ are not exclusive!)

Inclusion-Exclusion

The principle of inclusion-exclusion generalizes to three events:

$$\begin{array}{c}P[A\cup B\cup C]=\\ \\ P[A]+P[B]+P[C]-P[A\cap B]-P[A\cap C]-P[B\cap C]+P[A\cap B\cap C]\end{array}$$

The same pattern works for any number of events!

The pattern goes: “include singles” then “exclude doubles” then “include triples” then …

Include, exclude, include, exclude, include, …

Link to original

Illustration

Example - iPhones and iPads

iPhones and iPads

At Mr. Jefferson’s University, 25% of students have an iPhone, 30% have an iPad, and 60% have neither.

What is the probability that a randomly chosen student has some iProduct? (Q1)

What about both? (Q2)

Solution

(1) Set up the probability model:

A student is chosen at random:

Outcomes are chosen students.

The sample space $S$ is the set of all students. Events are subsets of $S$.

Write $O=\text{“has iPhone”}$ and $A=\text{“has iPad”}$ (regarding the chosen student).

All students are equally likely to be chosen. Therefore $P[E]=\frac{|E|}{|S|}$ for any event $E$. Therefore $P[O]=0.25$ and $P[A]=0.30$.

Furthermore, $P[O^c{A}^c]=0.60$. This states that 60% have “not iPhone AND not iPad”.

---

(2) Define the desired event:

Q1: $\text{desired event}=O\cup A$

Q2: $\text{desired event}=OA$

---

(3) Compute the probabilities:

We do not know that $O$ and $A$ are exclusive.

We could try inclusion-exclusion:

$$P[O\cup A]=P[O]+P[A]-P[OA]$$

We know $P[O]=0.25$ and $P[A]=0.30$. So this formula, with given data, RELATES Q1 and Q2. It does not solve either one by itself.

We have not yet used the information that $P[O^c{A}^c]=0.60$.

To use this, simplify it with De Morgan’s Laws:

$$P[O^c{A}^c]\gg \gg P[(O\cup A{)}^c]\gg \gg 1-P[O\cup A]$$

Therefore:

$$P[O^c{A}^c]=0.60\gg \gg P[O\cup A]=0.40$$

We have answered Q1. Recall that inclusion-exclusion relates Q1 and Q2 and solve to answer Q2:

$$\begin{array}{c}P[O\cup A]=P[O]+P[A]-P[OA]\\ \\ \gg \gg 0.40=0.25+0.30-P[OA]\\ \\ \gg \gg P[OA]=0.15\end{array}$$Link to original

Example - Lucia is Host or Player

Lucia is Host or Player

The professor chooses three students at random for a game in a class of 40, one to be Host, one to be Player, one to be Judge. What is the probability that Lucia is either Host or Player?

Solution

(1) Set up the probability model:

Label the students $1$ to $40$. Write $L$ for Lucia’s number.

Outcomes: assignments such as $(H,P,J)=(2,5,8)$. These are ordered triples with distinct entries in $1,2,\dots ,40$.

Sample space: $S$ is the collection of all such distinct triples

Events: any subset of $S$

Probability measure: assume all outcomes are equally likely, so $P[(i,j,k)]=P[(r,l,p)]$ for all $i,j,k,r,l,p$

In total there are $40\cdot 39\cdot 38$ triples of distinct numbers.

Therefore $P[(i,j,k)]=\frac{1}{40\cdot 39\cdot 38}$ for any specific outcome $(i,j,k)$.

Therefore $P[A]=\frac{|A|}{40\cdot 39\cdot 38}$ for any event $A$. (Recall $|A|$ is the number of outcomes in $A$.)

---

(2) Define the desired event:

We want to find $P[\text{“Lucia is Host or Player”}]$. Define $A=\text{“Lucia is Host”}$ and $B=\text{“Lucia is Player”}$. Thus:

$$A=\{(L,j,k)\mid \text{any }j,k\},B=\{(i,L,k)\mid \text{any }i,k\}$$

So, in this notation, we seek $P[A\cup B]$.

---

(3) Compute the desired probability:

Importantly, $A\cap B=\varnothing$ (mutually exclusive). There are no outcomes in $S$ in which Lucia is both Host and Player.

By additivity, we infer $P[A\cup B]=P[A]+P[B]$.

Now compute $P[A]$. There are $39\cdot 38$ ways to choose $j$ and $k$ from the students besides Lucia. Therefore $|A|=39\cdot 38$. Therefore:

$$P[A]\gg \gg \frac{|A|}{40\cdot 39\cdot 38}\gg \gg \frac{39\cdot 38}{40\cdot 39\cdot 38}\gg \gg \frac{1}{40}$$

Now compute $P[B]$. It is similar: $P[B]=\frac{1}{40}$.

Finally compute that $P[A]+P[B]=\frac{1}{20}$, so the answer is:

$$P[A\cup B]\gg \gg P[A]+P[B]\gg \gg \frac{1}{20}$$ Link to original

Conditional probability

Theory

Theory 1

Conditional probability

The conditional probability of “$B$ given $A$” is defined by:

$$P[B\mid A]=\frac{P[B\cap A]}{P[A]}$$

This conditional probability $P[B\mid A]$ represents the probability of event $B$ taking place given the assumption that $A$ took place. (All within the given probability model.)

By letting the actuality of event $A$ be taken as a fixed hypothesis, we can define a conditional probability measure by plugging events into the slot of $B$:

$$P[-\mid A]=\frac{P[-\cap A]}{P[A]}$$

It is possible to verify each of the Kolmogorov axioms for this function, and therefore $P[-\mid A]$ itself defines a bona fide probability measure.

Conditioning

What does it really mean?

Conceptually, $P[B\mid A]$ corresponds to creating a new experiment in which we run the old experiment and record data only those times that $A$ happened. Or, it corresponds to finding ourselves with knowledge or data that $A$ happened, and we seek our best estimates of the likelihoods of other events, based on our existing model and the actuality of $A$.

Mathematically, $P[B\mid A]$ corresponds to restricting the probability function to outcomes in $A$, and renormalizing the values (dividing by $P[A]$) so that the total probability of all the outcomes (in $A$) is now $1$.

The definition of conditional probability can also be turned around and reinterpreted:

Multiplication rule

$$P[AB]=P[A]\cdot P[B\mid A]$$

“The probability of $A$ AND $B$ equals the probability of $A$ times the probability of $B$-given-$A$.”

This principle generalizes to any events in sequence:

Generalized multiplication rule

$$P[A_1{A}_2{A}_3]=P[A_1]\cdot P[A_2\mid A_1]\cdot P[A_3\mid A_1{A}_2]$$ $$P[A_1\cdots A_n]=P[A_1]\cdot P[A_2\mid A_1]\cdot P[A_3\mid A_1{A}_2]\cdots P[A_n\mid A_1\cdots A_{n-1}]$$

The generalized rule can be verified like this. First substitute $A_2$ for $B$ and $A_1$ for $A$ in the original rule. Now repeat, substituting $A_3$ for $B$ and $A_1{A}_2$ for $A$ in the original rule, and combine with the first one, and you find the rule for triples. Repeat again with $A_4$ and $A_1{A}_2{A}_3$, combine with the triples, and you get quadruples.

Link to original

Illustration

Practice exercise

Simplifying conditionals inclusion

Let $A\subset B$. Simplify the following values:

$$P[A\mid B],P[A\mid B^c],P[B\mid A],P[B\mid A^c]$$

Solution

$P[A\mid B]$

• Definition of ‘conditional’: $P[A\mid B]=\frac{P[A\cap B]}{P[B]}$
• The problem assumes that $A\subset B$. Therefore $A\cap B=A$.
• Therefore, answer: $\frac{P[A]}{P[B]}$.

$P[A\mid B^c]$

• Definition of ‘conditional’: $P[A\mid B^c]=\frac{P[A\cap B^c]}{P[B^c]}$
• Since $A\subset B$, we know $A\cap B^c=\varnothing$.
• Therefore $P[A\cap B^c]=0$ and answer $=0$.

$P[B\mid A]$

• Definition of ‘conditional’: $P[B\mid A]=\frac{P[B\cap A]}{P[A]}$
• Since $A\subset B$, we have $B\cap A=A$.
• Therefore, answer $=1$.

$P[B\mid A^c]$

• Definition of ‘conditional’: $P[B\mid A^c]=\frac{P[B\cap A^c]}{P[A^c]}$
• There is no way to simplify further.
• We could write $P[A^c]=1-P[A]$ if desired.

Link to original

Example - Multiplication: draw two cards

Multiplication: draw two cards

Two cards are drawn from a standard deck (without replacement).

What is the probability that the first is a 3, and the second is a 4?

Solution

This “two-stage” experiment lends itself to a solution using the multiplication rule for conditional probability.

(1) Label events:

• Write $T$ for the event that the first card is a 3.
• Write $F$ for the event that the second card is a 4.

We seek $P[TF]$. We will use the multiplication rule:

$$P[TF]=P[T]\cdot P[F\mid T]$$

---

(2) Compute probabilities:

We know $P[T]=\frac{4}{52}$. (Does not depend on the second draw.)

For the conditional probability, note that if the first is a 3, then there are four remaining 4s and 51 remaining cards. Therefore:

$$P[F\mid T]=\frac{4}{51}$$

---

(3) Apply multiplication rule:

$$\begin{array}{c}P[TF]=P[T]\cdot P[F\mid T]\\ \\ \\ P[TF]=\frac{4}{52}\cdot \frac{4}{51}\gg \gg \frac{4}{13\cdot 51}\end{array}$$ Link to original

Example - Flip a coin, then roll dice

Multiplication: flip a coin, then roll dice

Flip a coin. If the outcome is heads, roll two dice and add the numbers. If the outcome is tails, roll a single die and take that number. What is the probability of getting a tails AND a number at least 3?

Solution

(1) This “two-stage” experiment lends itself to a solution using the multiplication rule for conditional probability.

Label the events of interest.

Let $H$ and $T$ be the events that the coin showed heads and tails, respectively.

Let $A_1,\dots ,A_{12}$ be the events that the final number is $1,\dots ,12$, respectively.

The value we seek is $P[T{A}_{\ge 3}]$.

---

(2) Observe known (conditional) probabilities.

We know that $P[H]=\frac{1}{2}$ and $P[T]=\frac{1}{2}$.

We know that $P[A_5\mid T]=\frac{1}{6}$, for example, or that $P[A_2\mid H]=\frac{1}{36}$.

---

(3) Apply multiplication rule:

$$P[T{A}_{\ge 3}]=P[T]\cdot P[A_{\ge 3}\mid T]$$

We know $P[T]=\frac{1}{2}$ and can see by counting that $P[A_{\ge 3}\mid T]=\frac{2}{3}$.

Therefore $P[T{A}_{\ge 3}]=\frac{1}{3}$.

Link to original

Example - Coin flipping: at least 2 heads

Coin flipping: at least 2 heads

Flip a fair coin 4 times and record the outcomes as sequences, like HHTH.

Let $A_{\ge 2}$ be the event that there are at least two heads, and $A_{\ge 1}$ the event that there is at least one heads.

First let’s calculate $P[A_{\ge 2}]$.

Define $A_2$, the event that there were exactly 2 heads, and $A_3$, the event of exactly 3, and $A_4$ the event of exactly 4. These events are exclusive, so:

$$P[A_{\ge 2}]=P[A_2\cup A_3\cup A_4]\gg \gg P[A_2]+P[A_3]+P[A_4]$$

Each term on the right can be calculated by counting:

$$\begin{array}{rl}P[A_2]& =\frac{|A_2|}{2^4}\gg \gg \frac{\left(\genfrac{}{}{0ex}{}{4}{2}\right)}{16}\gg \gg \frac{6}{16}\\ & \\ P[A_3]& =\frac{|A_3|}{2^4}\gg \gg \frac{\left(\genfrac{}{}{0ex}{}{4}{3}\right)}{16}\gg \gg \frac{4}{16}\\ & \\ P[A_4]& =\frac{|A_4|}{2^4}\gg \gg \frac{\left(\genfrac{}{}{0ex}{}{4}{4}\right)}{16}\gg \gg \frac{1}{16}\end{array}$$

Therefore, $P[A_{\ge 2}]=\frac{11}{16}$.

Now suppose we find out that “at least one heads definitely came up”. (Meaning that we know $A_{\ge 1}$.) For example, our friend is running the experiment and tells us this fact about the outcome.

Now what is our estimate of likelihood of $A_{\ge 2}$?

The formula for conditioning gives:

$$P[A_{\ge 2}\mid A_{\ge 1}]=\frac{P[A_{\ge 2}\cap A_{\ge 1}]}{P[A_{\ge 1}]}$$

Now $A_{\ge 2}\cap A_{\ge 1}=A_{\ge 2}$. (Any outcome with at least two heads automatically has at least one heads.) We already found that $P[A_{\ge 2}]=\frac{11}{16}$. To compute $P[A_{\ge 1}]$ we simply add the probability $P[A_1]$, which is $\frac{4}{16}$, to get $P[A_{\ge 1}]=\frac{15}{16}$.

Therefore:

$$P[A_{\ge 2}\mid A_{\ge 1}]=\frac{11/16}{15/16}\gg \gg \frac{11}{15}$$ Link to original

Theory

Theory 2

Law of Total Probability: 2 cases

For any events $A$ and $B$:

$$P[B]=P[A]\cdot P[B\mid A]+P[A^c]\cdot P[B\mid A^c]$$

This rule can also be called “Division into Cases.”

Interpretation: event $B$ may be divided along the lines of $A$, with some of $P[B]$ coming from the part in $A$ and the rest from the part in $A^c$.

Total Probability - Explanation

• First divide $B$ itself into parts in and out of $A$:

$$B=B\cap A\bigcup B\cap A^c$$

• These parts are exclusive, so in probability we have:

$$P[B]=P[BA]+P[B{A}^c]$$

• Use the Multiplication rule to break up $P[BA]$ and $P[B{A}^c]$:

$$\begin{array}{rl}P[BA]\gg \gg & P[A]\cdot P[B\mid A]\\ & \\ P[B{A}^c]\gg \gg & P[A^c]\cdot P[B\mid A^c]\end{array}$$

• Now substitute in the prior formula:

$$P[B]\gg \gg P[BA]+P[B{A}^c]\gg \gg P[A]\cdot P[B\mid A]+P[A^c]\cdot P[B\mid A^c]$$

This law can be generalized to any partition of the sample space $S$. A partition is a collection of events $A_i$ which are mutually exclusive and jointly exhaustive:

$$A_i\cap A_j=\varnothing ,\bigcup _i{A}_i=S$$

The generalized formulation of Total Probability for a partition is:

Law of Total Probability: $n$ cases

For a partition $A_i$ of the sample space $S$:

$$P[B]=\sum _{i=1}^{n}P[A_i]\cdot P[B\mid A_i]$$

By setting $n=2$ with $A_1=A$ and $A_2=A^c$, we recover the $2$-case Law from the $n$-case version of the Law.

Link to original

Illustration

Practice exercise

Marble transferred, marble drawn

Setup:

• Bin 1 holds five red and four green marbles.
• Bin 2 holds four red and five green marbles.

Experiment:

• You take a random marble from Bin 1 and put it in Bin 2 and shake Bin 2.
• Then you draw a random marble from Bin 2 and look at it.

What is the probability that the marble you look at is red?

Solution

(1) Label events:

• Event $T_R$: a red marble is transferred.
• Event $T_G$: a green marble is transferred.
• Event $D_R$: a red marble is drawn from Bin 2.
• Event $D_G$: a green marble is drawn from Bin 2.

Answer will be $P[D_R]$.

---

(2) Apply Division into Cases:

General formula: $P[B]=P[A]\cdot P[B\mid A]+P[A^c]\cdot P[B\mid A^c]$

Insert our labels, $A\to T_R$ and $B\to D_R$ and $A^c\to T_G$. Obtain:

$$P[D_R]=P[T_R]\cdot P[D_R\mid T_R]+P[T_G]\cdot P[D_R\mid T_G]$$

---

(3) Plug in data and compute:

• Know $P[T_R]=5/9$.
• Know $P[T_G]=4/9$.
• Know $P[D_R\mid T_R]=5/10$.
• Know $P[D_R\mid T_G]=4/10$.

Therefore:

$$P[D_R]\gg \gg \frac{5}{9}\cdot \frac{5}{10}+\frac{4}{9}\cdot \frac{4}{10}\gg \gg \frac{41}{90}$$

Link to original

Bayes’ Theorem

Theory

Theory 1

Bayes’ Theorem

For any events $A$ and $B$:

$$P[B\mid A]=P[B]\cdot \frac{P[A\mid B]}{P[A]}$$

Note: Bayes’ Theorem is sometimes called Bayes’ Rule.

Bayes’ Theorem - Derivation

Start with the observation that $AB=BA$, in other words event “$A$ AND $B$” equals event “$B$ AND $A$”.

Apply the multiplication rule to each product:

$$\begin{array}{rl}P[AB]& =P[A]\cdot P[B\mid A]\\ & \\ P[BA]& =P[B]\cdot P[A\mid B]\end{array}$$

Equate them and rearrange:

$$\begin{array}{rl}P[AB]=P[BA]\gg \gg & P[A]\cdot P[B\mid A]=P[B]\cdot P[A\mid B]\\ & \\ \gg \gg & P[B\mid A]=P[B]\cdot \frac{P[A\mid B]}{P[A]}\end{array}$$

The main application of Bayes’ Theorem is to calculate $P[A\mid B]$ when it is easy to calculate $P[B\mid A]$ from the problem setup. Often this occurs in multi-stage experiments where event $A$ describes outcomes of an intermediate stage.

Note: These lecture notes use alphabetical order $A$, $B$ as a mnemonic for temporal or logical order, i.e. that $A$ comes first in time, or that $A$ is the prior conditional from which it is easy to calculate $B$.

Link to original

Illustration

Example - Bayes’ Theorem - COVID tests

Bayes’ Theorem: COVID tests

Assume that 0.5% of people have COVID. Suppose a COVID test gives a (true) positive on 96% of patients who have COVID, but gives a (false) positive on 2% of patients who do not have COVID. Bob tests positive. What is the probability that Bob has COVID?

Solution

(1) Label events:

• Event $A_P$: Bob is actually positive for COVID
• Event $A_N$: Bob is actually negative; note $A_N=A_P^c$
• Event $T_P$: Bob tests positive
• Event $T_N$: Bob tests negative; note $T_N=T_P^c$

---

(2) Identify known data:

• Know: $P[T_P\mid A_P]=96\%$
• Know: $P[T_P\mid A_N]=2\%$
• Know: $P[A_P]=0.5\%$ and therefore $P[A_N]=99.5\%$

We seek: $P[A_P\mid T_P]$

---

(3) Translate Bayes’ Theorem:

Set $A=T_P$ and $B=A_P$ in Bayes’:

$$P[A_P\mid T_P]=P[A_P]\cdot \frac{P[T_P\mid A_P]}{P[T_P]}$$

We know all values on the right except $P[T_P]$

---

(4) Denominator: apply Total Probability (Division into Cases):

Observe that $T_P\cap A_P$ and $T_P\cap A_N$ are exclusive events, and that:

$$T_P=T_P\cap A_P\bigcup T_P\cap A_N.$$

Therefore:

$$P[T_P]=P[A_P]\cdot P[T_P\mid A_P]+P[A_N]\cdot P[T_P\mid A_N]$$

Plug in data and compute:

$$\gg \gg P[T_P]=\frac{5}{1000}\cdot \frac{96}{100}+\frac{995}{1000}\cdot \frac{2}{100}\gg \gg \approx 0.0247$$

---

(5) Plug in and compute:

$$\begin{array}{c}P[A_P\mid T_P]=P[A_P]\cdot \frac{P[T_P\mid A_P]}{P[T_P]}\\ \\ \gg \gg 0.96\cdot \frac{0.005}{0.0247}\gg \gg \approx 19\%\end{array}$$

Intuition - COVID testing

Some people find this low number surprising. In order to repair your intuition, think about it like this: roughly 2.5% of tests are positive, with roughly 2% coming from false positives, and roughly 0.5% from true positives. Only $1/5$ of all the positive results are true ones!

(This rough approximation assumes that $96\%\approx 100\%$.)

If two tests both come back positive, the odds of COVID are now 98%.

If only people with symptoms are tested, so that, say, 20% of those tested have COVID, that is, $P[A_P\mid T_P]=20\%$, then one positive test implies a COVID probability of 92%.

Link to original

Practice exercise

Inferring bin from marble

There are marbles in bins in a room:

• Bin 1 holds 7 red and 5 green marbles.
• Bin 2 holds 4 red and 3 green marbles.

Your friend goes in the room, shuts the door, and selects a random bin, then draws a random marble. (Equal odds for each bin, then equal odds for each marble in that bin.) He comes out and shows you a red marble.

What is the probability that this red marble was taken from Bin 1?

Solution

(1) Label events:

• Event $B_1$: friend chooses Bin 1.
• Event $B_2$: friend chooses Bin 2.
• Event $R$: friend draws a red marble.
• Event $G$: friend draws a green marble.

Answer will be $P[B_1\mid R]$.

---

(2) Identify knowns:

• Know $P[R\mid B_1]=7/12$.
• Know $P[G\mid B_1]=5/12$.
• Know $P[R\mid B_2]=4/7$.
• Know $P[G\mid B_2]=3/7$.
• Know $P[B_1]=P[B_2]=1/2$.

---

(3) Apply Bayes’ Theorem for $P[B_1\mid R]$::

$$P[B_1\mid R]=P[R\mid B_1]\cdot \frac{P[B_1]}{P[R]}$$

Division into Cases for the denominator:

$$P[R]=P[B_1]\cdot P[R\mid B_1]+P[B_2]\cdot P[R\mid B_2]$$

---

(3) Plug in data and compute:

$$\begin{array}{c}P[R]\gg \gg \frac{1}{2}\cdot \frac{7}{12}+\frac{1}{2}\cdot \frac{4}{7}\gg \gg \frac{97}{168}\\ \\ P[B_1\mid R]\gg \gg \frac{7}{12}\cdot \frac{1/2}{97/168}\gg \gg \frac{49}{97}\end{array}$$

Link to original

Independence

Theory

Theory 1

Two events are independent when information about one of them does not change our probability estimate for the other.

Independence

Events $A$ and $B$ are independent when these (logically equivalent) equations hold:

• $P[B\mid A]=P[B]$
• $P[A\mid B]=P[A]$
• $P[BA]=P[B]\cdot P[A]$

Note that the last equation is symmetric in $A$ and $B$:

• Check: $BA=AB$ and $P[B]\cdot P[A]=P[A]\cdot P[B]$
• This symmetric version is the preferred definition of the concept of independence.

Multiple-independence

A collection of events $A_1,\dots ,A_n$ is mutually independent when every subcollection $A_{i_1},\dots ,A_{i_k}$ satisfies:

$$P[A_{i_1}\cdots A_{i_k}]=P[A_{i_1}]\cdots P[A_{i_k}]$$

A potentially weaker condition for a collection $A_1,\dots ,A_n$ is called pairwise independence, which holds when all 2-member subcollections are independent:

$$P[A_i{A}_j]=P[A_i]\cdot P[A_j]\text{for all}i\ne j$$

One could also define $3$-member independence, or $n$-member independence. Plain ‘independence’ means any-member independence.

Link to original

Illustration

Practice exercise

Independence and complements

Prove that these are logically equivalent statements:

• $A$ and $B$ are independent
• $A$ and $B^c$ are independent
• $A^c$ and $B^c$ are independent

Make sure you demonstrate both directions of each equivalency.

Solution

(1) Assume $P[AB]=P[A]\cdot P[B]$ and show $P[A{B}^c]=P[A]\cdot P[B^c]$:

Divide $A$ into the $B$ cases:

$$P[A]=P[AB]+P[A{B}^c]$$

Apply the assumption:

$$\gg \gg P[A]=P[A]\cdot P[B]+P[A{B}^c]$$

Algebra:

$$\gg \gg P[A](1-P[B])=P[A{B}^c]$$

Negation rule:

$$\gg \gg P[A]\cdot P[B^c]=P[A{B}^c]$$

---

(2) Assume $P[A{B}^c]=P[A]\cdot P[B^c]$ and show $P[AB]=P[A]\cdot P[B]$:

Reason the same way as above, but apply the new assumption to break up the second term instead of the first.

---

(3) Show that $P[A{B}^c]=P[A]\cdot P[B^c]$ and $P[A^c{B}^c]=P[A^c]\cdot P[B^c]$ are equivalent:

To do this, simply notice that in the first equivalence we can replace $A$ with $B^c$ and $B$ with $A$. Use $AB=BA$ too.

Link to original

Example - Checking independence by hand

Independence by hand: red and green marbles

A bin contains 4 red and 7 green marbles. Two marbles are drawn.

Let $R_1$ be the event that the first marble is red, and let $G_2$ be the event that the second marble is green.

(a) Show that $R_1$ and $G_2$ are independent if the marbles are drawn with replacement.

(b) Show that $R_1$ and $G_2$ are not independent if the marbles are drawn without replacement.

Solution

(a) With replacement.

Identify knowns:

• Know: $P[R_1]=\frac{4}{11}$
• Know: $P[G_2]=\frac{7}{11}$

Now compute both sides of independence relation:

$$P[R_1{G}_2]=P[R_1]\cdot P[G_2]$$

The right side is $\frac{4}{11}\cdot \frac{7}{11}$.

For $P[R_1{G}_2]$, we have $4\cdot 7$ ways to get $R_1{G}_2$, and $11^2$ total outcomes. So left side is $\frac{4\cdot 7}{11^2}$, which equals the right side.

---

(b) Without replacement. This is a bit harder.

(1) Identify knowns:

Know: $P[R_1]=\frac{4}{11}$ and therefore $P[R_1^c]=\frac{7}{11}$

We seek: $P[G_2]$ and $P[R_1{G}_2]$

---

(2) Find $P[G_2]$ using Total Probability (Division into Cases):

$$\begin{array}{c}{G}_2=G_2\cap R_1\bigcup G_2\cap R_1^c\\ \\ \gg \gg P[G_2]=P[R_1]\cdot P[G_2\mid R_1]+P[R_1^c]\cdot P[G_2\mid R_1^c]\end{array}$$

Find RHS factors by counting, then compute:

$$\gg \gg P[G_2]=\frac{4}{11}\cdot \frac{7}{10}+\frac{7}{11}\cdot \frac{6}{10}\gg \gg \frac{70}{110}$$

---

(3) Find $P[R_1{G}_2]$ using multiplication rule:

$$P[R_1{G}_2]=P[R_1]\cdot P[G_2\mid R_1]\gg \gg \frac{4}{11}\cdot \frac{7}{10}\gg \gg \frac{28}{110}$$

---

(4) Compare both sides:

Left side: $P[R_1{G}_2]=\frac{28}{110}$.

Right side:

$$P[R_1]\cdot P[G_2]=\frac{4}{11}\cdot \frac{70}{110}=\frac{28}{121}$$

But $\frac{28}{110}\ne \frac{28}{121}$ so $P[R_1{G}_2]\ne P[R_1]\cdot P[G_2]$ and they are not independent.

Link to original

Tree diagrams

Theory

Theory 1

A tree diagram depicts the components of a multi-stage experiment. Nodes represent sources of randomness.

An outcome of the experiment is represented by a complete path taken from the root (left-most node, only one option) to a leaf (right-most node, many options). The branch chosen at a given node represents the outcome of a “sub-experiment.” So a complete path encodes the outcomes of all sub-experiments along the way.

Each branch emanating from a node is labeled with a probability value. This is the probability that the sub-experiment of that node has the outcome of that branch. (In the example, $0.8=P[A\mid B_1]$.) This is also the conditional probability of the branch’s right node, given its left node as known.

Therefore, branch values from any given node must sum to 1.

The probability of a given outcome is the product of the probabilities along each branch of the path from the root to that outcome.

For example, for outcome $A{B}_1$, we have $P[A{B}_1]=P[A]\cdot P[B_1\mid A]$.

Generally, remember that

$$\begin{array}{rl}P[ABCD]& =P[ABC]\cdot P[D\mid ABC]\\ & \\ & =P[AB]\cdot P[C\mid AB]\cdot P[D\mid ABC]\\ & \\ & =P[A]\cdot P[B\mid A]\cdot P[C\mid AB]\cdot P[D\mid ABC]\end{array}$$

This overall outcome probability may be written at the final leaf. (Not to be confused with the branch value of the last branch.)

One can also use a tree diagram to remember quickly how to calculate certain probabilities.

For example, what is $P[A]$ in the diagram?

• Answer: add up the path probabilities for all paths terminating in $A$. We obtain:

$$0.24+0.36+0.18\gg \gg 0.78$$

For example, what is $P[B_1\mid N]$?

• Answer: divide the leaf probability of $B_{1}N$ by the total probability of $N$. We obtain:

$$P[B_1\mid N]=\frac{0.06}{0.06+0.04+0.12}\approx 0.27$$Link to original

Illustration

Example - Tree diagrams: Marble transferred, marble drawn

Marble transferred, marble drawn

Setup:

• Bin 1 holds five red and four green marbles.
• Bin 2 holds four red and five green marbles.

Experiment:

• You take a random marble from Bin 1 and put it in Bin 2 and shake Bin 2.
• Then you draw a random marble from Bin 2 and look at it.

Questions:

(a) What is the probability you draw a red marble?

(b) Supposing that you drew a red marble, what is the probability that a red marble was transferred?

Solution

Construct the tree diagram:

Identify sub-experiments, label events, compute probabilities:

---

(a) Compute $P[D_R]$:

Add up leaf numbers for $D_R$ at leaf:

$$P[D_R]=\frac{25}{90}+\frac{16}{90}=\frac{41}{90}$$

---

(b) Compute $P[T_R\mid D_R]$:

Conditional probability definition:

$$\begin{array}{c}P[T_R\mid D_R]=\frac{P[T_R{D}_R]}{P[D_R]}\\ \\ \gg \gg \frac{25/90}{41/90}\gg \gg \frac{25}{41}\end{array}$$

Interpretation: value of desired path over value of all possible paths.

Link to original