Chapter 06 - Covariance and Conditional Distributions

Expectation for two variables

Theory

Theory 1

Expectation for a function on two variables

Discrete case:

$$E[g(X,Y)]=\sum _{k,\ell }g(k,\ell )P_{X,Y}(k,\ell )\text{(sum over possible values)}$$

Continuous case:

$$E[g(X,Y)]={\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }g(x,y)f_{X,Y}(x,y)dxdy$$

These formulas are not trivial to prove, and we omit the proofs. (Recall the technical nature of the proof we gave for $E[g(X)]$ in the discrete case.)

Expectation sum rule

Suppose $X$ and $Y$ are any two random variables on the same probability model.

Then:

$$E[X+Y]=E[X]+E[Y]$$

We already know that expectation is linear in a single variable: $E[aX+b]=aE[X]+b$.

Therefore this two-variable formula implies:

$$E[aX+bY+c]=aE[X]+bE[Y]+c$$

Expectation product rule: independence

Suppose that $X$ and $Y$ are independent.

Then we have:

$$E[XY]=E[X]E[Y]$$

Extra - Proof: Expectation sum rule, continuous case

Suppose $f_X$ and $f_Y$ give marginal PDFs for $X$ and $Y$, and $f_{X,Y}$ gives their joint PDF.

Then:

$$\begin{array}{rl}E[X+Y]& \gg \gg {\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }(x+y)f_{X,Y}(x,y)dxdy\\ & \\ & \gg \gg {\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }x{f}_{X,Y}dxdy+{\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }y{f}_{X,Y}dxdy\\ & \\ & \gg \gg {\int }_{-\infty }^{+\infty }x\left({\int }_{-\infty }^{+\infty }{f}_{X,Y}dy\right)dx+{\int }_{-\infty }^{+\infty }y\left({\int }_{-\infty }^{+\infty }{f}_{X,Y}dx\right)dy\\ & \\ & \gg \gg {\int }_{-\infty }^{+\infty }x{f}_X(x)dx+{\int }_{-\infty }^{+\infty }y{f}_Y(y)dy\\ & \\ & \gg \gg E[X]+E[Y]\end{array}$$

Observe that this calculation relies on the formula for $E[g(X,Y)]$, specifically with $g(x,y)=x+y$.

Extra - Proof: Expectation product rule

$$\begin{array}{rl}E[XY]& \gg \gg {\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }(xy)f_{X,Y}(x,y)dxdy\\ & \\ & \gg \gg {\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }(xy)f_X(x)f_Y(y)dxdy\\ & \\ & \gg \gg {\int }_{-\infty }^{+\infty }x{f}_X(x)dx{\int }_{-\infty }^{+\infty }y{f}_Y(y)dy\\ & \\ & \gg \gg E[X]E[Y]\end{array}\text{\hspace{1em}(independence)}$$

Link to original

Illustration

$E[X^2+Y]$ from joint PMF chart

Expectation of X squared plus Y from joint PMF chart

Suppose the joint PMF of $X$ and $Y$ is given by this chart:

$Y\downarrow X\to$	$1$	$2$
$-1$	0.2	0.2
$0$	0.35	0.1
$1$	0.05	0.1

Define $W=X^2+Y$. Find the expectation $E[W]$.

Solution

First compute the values of $W$ for each pair $(X,Y)$ in the chart:

$Y\downarrow X\to$	$1$	$2$
$-1$	0	3
$0$	1	4
$1$	2	5

Now take the sum, weighted by probabilities:

$$\begin{array}{c}0\cdot (0.2)+3\cdot (0.2)+1\cdot (0.35)\\ +4\cdot (0.1)+2\cdot (0.05)+5\cdot (0.1)\end{array}\gg \gg 1.95=E[W]$$ Link to original

Exercise - Understanding expectation for two variables

Understanding expectation for two variables

Suppose you know only that $X\sim \mathrm{Geo}(p)$ and $Y\sim \mathrm{Bin}(n,q)$.

Which of the following can you calculate?

$$E[X+Y],E[XY],E[X^2+Y^2],E[(X+Y{)}^2]$$Link to original

$E[Y]$ two ways, and $E[XY]$, from joint density

Expectation of Y two ways and Expectation of XY from joint density

Suppose $X$ and $Y$ are random variables with the following joint density:

$$f_{X,Y}(x,y)=\{\begin{array}{ll}\frac{3}{16}x{y}^2& x,y\in [0,2]\\ 0& \text{ otherwise }\end{array}$$

(a) Compute $E[Y]$ using two methods.

(b) Compute $E[XY]$.

Solution

(a)

(1) Method One: via marginal PDF $f_Y(y)$:

$$f_Y(y)={\int }_0^2\frac{3}{16}x{y}^{2}dx\gg \gg \{\begin{array}{ll}\frac{3}{8}{y}^2& y\in [0,2]\\ 0& \text{otherwise}\end{array}$$

Then expectation:

$$E[Y]={\int }_0^{2}y{f}_Y(y)dy\gg \gg {\int }_0^2\frac{3}{8}{y}^{3}dy\gg \gg 3/2$$

(2) Method Two: directly, via two-variable formula:

$$E[Y]={\int }_0^2{\int }_0^{2}y\cdot \frac{3}{16}x{y}^{2}dydx\gg \gg {\int }_0^2\frac{3}{4}xdx\gg \gg 3/2$$

(b) Directly, via two-variable formula:

$$\begin{array}{c}E[XY]={\int }_0^2{\int }_0^{2}xy\cdot \frac{3}{16}x{y}^{2}dydx\\ \\ \gg \gg {\int }_0^2\frac{3}{4}{x}^{2}dx\gg \gg 2\end{array}$$Link to original

Covariance and correlation

Theory

Theory 1

Write ${\mu }_X=E[X]$ and ${\mu }_Y=E[Y]$.

Observe that the random variables $X-{\mu }_X$ and $Y-{\mu }_Y$ are “centered at zero,” meaning that $E[X-{\mu }_X]=0=E[Y-{\mu }_Y]$.

Covariance

Suppose $X$ and $Y$ are any two random variables on a probability model. The covariance of $X$ and $Y$ measures the typical synchronous deviation of $X$ and $Y$ from their respective means.

Then the defining formula for covariance of $X$ and $Y$ is:

$$\mathrm{Cov}[X,Y]=E[(X-{\mu }_X)(Y-{\mu }_Y)]$$

There is also a shorter formula:

$$\mathrm{Cov}[X,Y]=E[XY]-E[X]E[Y]$$

To derive the shorter formula, first expand the product $(X-{\mu }_X)(Y-{\mu }_Y)$ and then apply linearity.

Notice that covariance is always symmetric:

$$\mathrm{Cov}[X,Y]=\mathrm{Cov}[Y,X]$$

The self covariance equals the variance:

$$\mathrm{Cov}[X,X]=\mathrm{Var}[X]$$

The sign of $\mathrm{Cov}[X,Y]$ reveals the correlation type between $X$ and $Y$:

Correlation	Sign
Positively correlated	$\mathrm{Cov}[X,Y]>0$
Negatively correlated	$\mathrm{Cov}[X,Y]<0$
Uncorrelated	$\mathrm{Cov}[X,Y]=0$

Correlation coefficient

Suppose $X$ and $Y$ are any two random variables on a probability model.

Their correlation coefficient is a rescaled version of covariance that measures the synchronicity of deviations:

$$\rho [X,Y]=\frac{\mathrm{Cov}[X,Y]}{{\sigma }_X{\sigma }_Y}=\mathrm{Cov}\left[\frac{X}{{\sigma }_X},\frac{Y}{{\sigma }_Y}\right]$$

The rescaling ensures:

$$-1\le {\rho }_{X,Y}\le +1$$

Covariance depends on the separate variances of $X$ and $Y$ as well as their relationship.

Correlation coefficient, because we have divided out ${\sigma }_X{\sigma }_Y$, depends only on their relationship.

Link to original

Illustration

Covariance from PMF chart

Covariance from PMF chart

Suppose the joint PMF of $X$ and $Y$ is given by this chart:

$Y\downarrow X\to$	$1$	$2$
$-1$	0.2	0.2
$0$	0.35	0.1
$1$	0.05	0.1

Find $\mathrm{Cov}[X,Y]$.

Solution

We need $E[X]$ and $E[Y]$ and $E[XY]$.

$$E[X]=1\cdot (0.2+0.35+0.05)+2\cdot (0.2+0.1+0.1)\gg \gg 1.4$$ $$\begin{array}{c}E[Y]=-1\cdot (0.2+0.2)+0\cdot (0.35+0.1)+1\cdot (0.05+0.1)\\ \\ \gg \gg -0.25\end{array}$$ $$E[XY]=-1\cdot (0.2)-2\cdot (0.2)+0+1\cdot (0.05)+2\cdot (0.1)\gg \gg -0.35$$

Therefore:

$$\begin{array}{c}\mathrm{Cov}[X,Y]=E[XY]-E[X]E[Y]\\ \\ \gg \gg -0.35-(1.4)(-0.25)\gg \gg 0\end{array}$$ Link to original

Theory

Theory 2

Covariance bilinearity

Given any three random variables $X$, $Y$, and $Z$, we have:

$$\begin{array}{rl}\mathrm{Cov}[X+Y,Z]& =\mathrm{Cov}[X,Z]+\mathrm{Cov}[Y,Z]\\ & \\ \mathrm{Cov}[Z,X+Y]& =\mathrm{Cov}[Z,X]+\mathrm{Cov}[Z,Y]\end{array}$$

Covariance and correlation: shift and scale

Covariance scales with each input, and ignores shifts:

$$\mathrm{Cov}[aX+b,Y]=a\mathrm{Cov}[X,Y]=\mathrm{Cov}[X,aY+b]$$

Whereas shift or scale in correlation only affects the sign:

$$\rho [aX+b,Y]=\mathrm{sign}(a)\rho [X,Y]=\rho [X,aY+b]$$

Extra - Proof of covariance bilinearity

$$\begin{array}{rl}\mathrm{Cov}[X+Y,Z]& \gg \gg E[(X+Y-({\mu }_X+{\mu }_Y))(Z-{\mu }_Z)]\\ & \\ & \gg \gg E[(X-{\mu }_X+Y-{\mu }_Y)(Z-{\mu }_Z)]\\ & \\ & \gg \gg E[(X-{\mu }_X)(Z-{\mu }_Z)]+E[(Y-{\mu }_Y)(Z-{\mu }_Z)]\\ & \\ & \gg \gg \mathrm{Cov}[X,Z]+\mathrm{Cov}[Y,Z]\end{array}$$

Extra - Proof of covariance shift and scale rule

$$\begin{array}{rl}\mathrm{Cov}[aX+b,Y]& \gg \gg E[(aX+b)Y]-E[aX+b]E[Y]\\ & \\ & \gg \gg E[aXY+bY]-aE[X]E[Y]-E[b]E[Y]\\ & \\ & \gg \gg aE[XY]+bE[Y]-aE[X]E[Y]-bE[Y]\\ & \\ & \gg \gg a(E[XY]-E[X]E[Y])\end{array}$$

---

Independence implies zero covariance

Suppose that $X$ and $Y$ are any two random variables on a probability model.

If $X$ and $Y$ are independent, then:

$$\mathrm{Cov}[X,Y]=0$$

Proof:

We know both of these:

$$\begin{array}{c}E[XY]=E[X]E[Y]\\ \\ \mathrm{Cov}[X,Y]=E[XY]-{\mu }_X{\mu }_Y\end{array}\text{\hspace{1em}(independence)(shorter form)}$$

But $E[XY]=E[X]E[Y]={\mu }_X{\mu }_Y$, so those terms cancel and $\mathrm{Cov}[X,Y]=0$.

Sum rule for variance

Suppose that $X$ and $Y$ are any two random variables on a probability space.

Then:

$$\mathrm{Var}[X+Y]=\mathrm{Var}[X]+\mathrm{Var}[Y]+2\mathrm{Cov}[X,Y]$$

When $X$ and $Y$ are independent:

$$\mathrm{Var}[X+Y]=\mathrm{Var}[X]+\mathrm{Var}[Y]$$

Extra - Proof: Sum rule for variance

$$\begin{array}{rl}\mathrm{Var}[X+Y]& \gg \gg E\left[\right(X+Y-({\mu }_X+{\mu }_Y){)}^2]\\ & \\ & \gg \gg E\left[\right((X-{\mu }_X)+(Y-{\mu }_Y){)}^2]\\ & \\ & \gg \gg E[(X-{\mu }_X{)}^2+(Y-{\mu }_Y{)}^2+2(X-{\mu }_X)(Y-{\mu }_Y)]\\ & \\ & \gg \gg \mathrm{Var}[X]+\mathrm{Var}[Y]+2\mathrm{Cov}[X,Y]\end{array}$$

Extra - Proof that $-1\le \rho \le +1$

(1) Create standardizations:

$$\tilde{X}=\frac{X-{\mu }_X}{{\sigma }_X},\tilde{Y}=\frac{Y-{\mu }_Y}{{\sigma }_Y}$$

Now $\tilde{X}$ and $\tilde{Y}$ satisfy:

$$E[\tilde{X}]=0=E[\tilde{Y}]\text{and}\mathrm{Var}[\tilde{X}]=1=\mathrm{Var}[\tilde{Y}]$$

Observe that $\mathrm{Var}[W]\ge 0$ for any $W$. Variance can’t be negative.

---

(2) Apply the variance sum rule.

Apply to $\tilde{X}$ and $\tilde{Y}$:

$$0\le \mathrm{Var}[\tilde{X}+\tilde{Y}]=\mathrm{Var}[\tilde{X}]+\mathrm{Var}[\tilde{Y}]+2\mathrm{Cov}[\tilde{X},\tilde{Y}]$$

Simplify:

$$\begin{array}{c}\gg \gg 1+1+2\mathrm{Cov}[\tilde{X},\tilde{Y}]\ge 0\\ \\ \gg \gg 1+\mathrm{Cov}[\tilde{X},\tilde{Y}]\ge 0\end{array}$$

Notice effect of standardization:

$$\mathrm{Cov}[\tilde{X},\tilde{Y}]=\rho [X,Y]$$

Therefore $\rho [X,Y]\ge -1$.

---

(3) Modify and reapply variance sum rule.

Change to $\tilde{X}-\tilde{Y}$:

$$0\le \mathrm{Var}[\tilde{X}-\tilde{Y}]=\mathrm{Var}[\tilde{X}]+\mathrm{Var}[-\tilde{Y}]+2\mathrm{Cov}[\tilde{X},-\tilde{Y}]$$

Simplify:

$$\begin{array}{c}\gg \gg 1+1-2\mathrm{Cov}[\tilde{X},\tilde{Y}]\ge 0\\ \\ \gg \gg 1-\mathrm{Cov}[\tilde{X},\tilde{Y}]\ge 0\end{array}$$

Link to original

Illustration

Variance of sum of indicators

Variance of sum of indicators

An urn contains 3 red balls and 2 yellow balls.

Suppose 2 balls are drawn without replacement, and $X$ counts the number of red balls drawn.

Find $\mathrm{Var}[X]$.

Solution

Let $X_1$ indicate (one or zero) whether the first ball is red, and $X_2$ indicate whether the second ball is red, so $X=X_1+X_2$.

Then $X_1{X}_2$ indicates whether both drawn balls are red; so it is Bernoulli with success probability $\frac{3}{5}\frac{2}{4}=\frac{3}{10}$. Therefore $E[X_1{X}_2]=\frac{3}{10}$.

We also have $E[X_1]=E[X_2]=\frac{3}{5}$.

The variance sum rule gives:

$$\begin{array}{rl}\mathrm{Var}[X]& =\mathrm{Var}[X_1]+\mathrm{Var}[X_2]+2\mathrm{Cov}[X_1,X_2]\\ & \\ & \gg \gg E[X_1^2]-E[X_1{]}^2+E[X_2^2]-E[X_2{]}^2+2(E[X_1{X}_2]-E[X_1]E[X_2])\\ & \\ & \gg \gg \frac{3}{5}-{\left(\frac{3}{5}\right)}^2+\frac{3}{5}-{\left(\frac{3}{5}\right)}^2+2\left(\frac{3}{10}-\frac{3}{5}\cdot \frac{3}{5}\right)\gg \gg \frac{9}{25}\end{array}$$Link to original

Exercise - Covariance rules

Covariance rules

Simplify:

$$\mathrm{Cov}[2X+5Y+1,3Y+8W+X+9]$$ Link to original

Exercise - Independent variables are uncorrelated

Dependent but uncorrelated variables

Let $X$ be given with possible values $\{-1,0,+1\}$ and PMF given (uniformly) by $P_X(k)=1/3$ for all three possible $k$. Let $Y=X^2$.

Show that $X$ and $Y$ are dependent but uncorrelated.

Hint: To speed the calculation, notice that $X^3=X$.

Link to original

Review: True/False

TRUE or FALSE:

(a) Suppose $\mathrm{Cov}[X,Y]=0.05$. It is possible that ${\rho }_{X,Y}=0.05$.

(b) Suppose $\mathrm{Cov}[X,Y]=0.05$. It is possible that $X$ and $Y$ have a strong linear relationship.

(c) Suppose $\mathrm{Cov}[X,Y]=0$. It is possible that $X$ and $Y$ are not independent.

(d) Suppose X and Y are not independent. It is possible that $\mathrm{Cov}[X,Y]$ is equal to 0.

(e) Suppose X and Y are independent. $\mathrm{Cov}[X,Y]$ must be equal to 0.

Review: Conditional probability

Review

Recall some items related to conditional probability.

Conditioning definition:

$$P[-\mid A]=\frac{P[-\cap A]}{P[A]}$$

Multiplication rule:

$$P[AB]=P[B\mid A]P[A]$$

Division into Cases / Total Probability:

$$P[B]=P[B\mid A_1]P[A_1]+\cdots +P[B\mid A_n]P[A_n]$$Link to original

Conditional distribution

Theory

Theory 1

Conditional distribution - fixed event

Suppose $X$ is a random variable, and suppose $A\subset \mathbb{R}$. The distribution of $X$ conditioned on $A$ describes the probabilities of values of $X$ given knowledge that $X\in A$.

Discrete case:

$$P_{X|A}(k)=\{\begin{array}{ll}\frac{1}{P[A]}{P}_X(k)& k\in A\\ & \\ 0& k\notin A\end{array}$$

Continuous case:

$$f_{X|A}(x)=\{\begin{array}{ll}\frac{1}{P[A]}{f}_X(x)& x\in A\\ & \\ 0& x\notin A\end{array}$$

There is also a conditional CDF, of which this conditional PDF is the derivative:

$$F_{X|A}(x)=P[X\le x\mid A],f_{X|A}(x)=\frac{d}{dx}{F}_{X|A}(x)$$

The Law of Total Probability has versions for distributions:

$$\begin{array}{rl}{P}_X(k)& =P_{X|A_1}(k)P[A_1]+\cdots +P_{X|A_n}(k)P[A_n]\\ & \\ f_X(x)& =f_{X|A_1}(x)P[A_1]+\cdots +f_{X|A_n}(x)P[A_n]\end{array}$$

Conditional distribution - variable event

Suppose $X$ and $Y$ are any two random variables. The distribution of $X$ conditioned on $Y$ describes the probabilities of values of $X$ in terms of $y$, given knowledge that $Y=y$.

Discrete case:

$$\begin{array}{rl}{P}_{X|Y}(k|\ell )& =P[X=k\mid Y=\ell ]\\ & \\ & =\frac{P_{X,Y}(k,\ell )}{P_Y(\ell )}(\text{assuming }{P}_Y(\ell )\ne 0)\end{array}$$

Continuous case:

$$f_{X|Y}(x|y)=\frac{f_{X,Y}(x,y)}{f_Y(y)}(\text{assuming }{f}_Y(y)\ne 0)$$

Remember: $P_{X,Y}(k,\ell )$ is the probability that “$X=k$ and $Y=\ell$.”

Sometimes it is useful to have the formulas rewritten like this:

$$\begin{array}{rl}{P}_{X,Y}(k,\ell )& =P_{X|Y}(k|\ell )P_Y(\ell )\\ & \\ f_{X,Y}(x,y)& =f_{X|Y}(x|y)f_Y(y)\end{array}$$

Extra - Deriving $f_{X|Y}(x|y)$

The density $f_{X|Y}$ ought to be such that $f_{X|Y}(x|y)dx$ gives the probability of $X\in [x,x+dx]$, given knowledge that $Y\in [y,y+dy]$. Calculate this probability:

$$\begin{array}{c}P[x\le X\le x+dx|y\le Y\le y+dy]\\ \\ \gg \gg \frac{P[x\le X\le x+dx,y\le Y\le y+dy]}{P[y\le Y\le y+dy]}\\ \\ \gg \gg \frac{f_{X,Y}(x,y)dxdy}{f_Y(y)dy}\\ \\ \gg \gg \frac{f_{X,Y}(x,y)}{f_Y(y)}dx\end{array}$$

Link to original

Illustration

Example - Conditional PMF, variable event, via joint density

Conditional PMF, variable event, via joint density

Suppose $X$ and $Y$ have joint PMF given by:

$$P_{X,Y}(k,\ell )=\{\begin{array}{ll}\frac{k+\ell }{21}& k=1,2,3;\ell =1,2\\ 0& \text{otherwise}\end{array}$$

Find $P_{X|Y}(k|\ell )$ and $P_{Y|X}(\ell ,k)$.

Solution

Marginal PMFs:

$$P_X(k)=\frac{2k+3}{21},k=1,2,3$$ $$P_Y(\ell )=\frac{\ell +2}{7},\ell =1,2$$

Assuming $\ell =1$ or $2$, for each $k=1,2,3$ we have:

$$P_{X|Y}(k|\ell )=\frac{P_{X,Y}(k,\ell )}{P_Y(\ell )}\gg \gg \frac{k+\ell }{3\ell +6}$$

Assuming $k=1$, $2$, or $3$, for each $\ell =1,2$ we have:

$$P_{Y|X}(\ell |k)=\frac{P_{Y,X}(\ell ,k)}{P_X(k)}\gg \gg \frac{k+\ell }{2k+3}$$ Link to original

Conditional expectation

Theory

Theory 1

Expectation conditioned by a fixed event

Suppose $X$ is a random variable and $A\subset \mathbb{R}$. The expectation of $X$ conditioned on $A$ describes the typical value of $X$ given the hypothesis that $X\in A$ is known.

Discrete case:

$$\begin{array}{rl}E[X\mid A]& =\sum _{k}k{P}_{X|A}(k)\\ & \\ E[g(X)\mid A]& =\sum _{k}g(k)P_{X|A}(k)\end{array}$$

Continuous case:

$$\begin{array}{rl}E[X\mid A]& ={\int }_{-\infty }^{+\infty }x{f}_{X|A}(x)dx\\ & \\ E[g(X)\mid A]& ={\int }_{-\infty }^{+\infty }g(x)f_{X|A}(x)dx\end{array}$$

Conditional variance:

$$\mathrm{Var}[X\mid A]=E[(X-{\mu }_{X|A}{)}^2\mid A]=E[X^2\mid A]-{\mu }_{X|A}^2$$

Division into Cases / Total Probability applied to expectation:

$$E[X]=E[X\mid A_1]P[A_1]+\cdots +E[X\mid A_n]P[A_n]$$

Linearity of conditional expectation:

$$E[a{X}_1+b{X}_2+c\mid Y=y]=aE[X_1\mid Y=y]+bE[X_2\mid Y=y]+c$$

Extra - Proof: Division of Expectation into Cases

We prove the discrete case only.

1. Expectation formula:

$$E[X]=\sum _{k}k{P}_X(k)$$

2. Division into Cases for the PMF:

$$P_X(k)=\sum _{i=1}^n{P}_{X|A_i}(k)P[A_i]$$

3. Substitute in the formula for $E[X]$:

$$\begin{array}{c}\sum _{k}k{P}_X(k)\gg \gg \sum _{k}k\sum _{i=1}^n{P}_{X|A_i}(k)P[A_i]\\ \\ \gg \gg \sum _{i=1}^{n}P[A_i]\sum _{k}k{P}_{X|A_i}(k)\\ \\ \gg \gg \sum _{i=1}^{n}P[A_i]E[X\mid A_i]\end{array}$$

---

Expectation conditioned by a variable event

Suppose $X$ and $Y$ are any two random variables. The expectation of $X$ conditioned on $Y=y$ describes the typical of value of $X$ in terms of $y$, given the hypothesis that $Y=y$ is known.

Discrete case:

$$\begin{array}{rl}E[X\mid Y=y]& =\sum _{k}k{P}_{X|Y}(k|y)\text{(k over all poss. vals.)}\\ & \\ E[g(X,Y)\mid Y=y]& =\sum _{k}g(k,y)P_{X|Y}(k|y)\end{array}$$

Continuous case:

$$\begin{array}{rl}E[X\mid Y=y]& ={\int }_{-\infty }^{+\infty }x{f}_{X|Y}(x|y)dx\\ & \\ E[g(X,Y)\mid Y=y]& ={\int }_{-\infty }^{+\infty }g(x,y)f_{X|Y}(x|y)dx\end{array}$$

Link to original

Illustration

Example - Conditional PMF, fixed event, expectation

Conditional PMF, fixed event, expectation

Suppose $X$ measures the lengths of some items and has the following PMF:

$$P_X(k)=\{\begin{array}{ll}0.15& k=1,2,3,4\\ 0.1& k=5,6,7,8\\ 0& \text{ otherwise }\end{array}$$

Let $L=\text{“}X\ge 5\text{”}$, an event.

(a) Find the conditional PMF of $X$ given that $L$ is known.

(b) Find the conditional expected value and variance of $X$ given $L$.

Solution

(a)

Conditional PMF formula with $k\in L$ plugged in:

$$P_{X\mid L}(k)=\{\begin{array}{ll}\frac{P_X(k)}{P[L]}& k=5,6,7,8\\ 0& \text{ otherwise }\end{array}$$

Compute $P[L]$ by adding cases:

$$P[L]=\sum _{k=5}^8{P}_X(k)\gg \gg 0.4$$

Divide nonzero PMF entries by $0.1$:

$$P_{X\mid L}(k)=\{\begin{array}{ll}0.25& k=5,6,7,8\\ 0& \text{ otherwise }\end{array}$$

---

(b)

Find $E[X\mid L]$:

$$\begin{array}{c}E[X\mid L]=\sum _{k=5}^{8}k{P}_{X\mid L}(k)\\ \\ \gg \gg 5\cdot (0.25)+6\cdot (0.25)+7\cdot (0.25)+8\cdot (0.25)\\ \\ \gg \gg 6.5\min \end{array}$$

Find $E[X^2\mid L]$:

$$\begin{array}{c}E[X^2\mid L]=\sum _{k=5}^8{k}^2{P}_{X\mid L}(k)\\ \\ \gg \gg 5^2\cdot (0.25)+6^2\cdot (0.25)+7^2\cdot (0.25)+8^2\cdot (0.25)\\ \\ \gg \gg 43.5{\min }^2\end{array}$$

Find $\mathrm{Var}[X\mid L]$ using “short form” with conditioning:

$$\mathrm{Var}[X\mid L]=E[X^2\mid L]-E[X\mid L{]}^2\gg \gg 1.25{\min }^2$$ Link to original

Illustration

Example - Conditional expectations from joint density

Conditional expectations from joint density

Suppose $X$ and $Y$ are random variables with joint density given by:

$$f_{X,Y}(x,y)=\{\begin{array}{ll}\frac{1}{y}{e}^{-x/y}{e}^{-y}& x,y\in (0,\infty )\\ 0& \text{otherwise}\end{array}$$

Find $E[X\mid Y=y]$.

Solution

(1) Derive the marginal density $f_Y(y)$:

$$\begin{array}{c}{f}_Y(y)\gg \gg {\int }_0^{+\infty }\frac{1}{y}{e}^{-x/y}{e}^{-y}dx\\ \\ \gg \gg -e^{-x/y}{e}^{-y}{|}_{x=0}^{\infty }\gg \gg e^{-y}\end{array}$$

---

(2) Use $f_Y(y)$ to compute $f_{X|Y}(x|y)$:

$$\begin{array}{c}{f}_{X|Y}(x|y)\gg \gg \frac{f_{X,Y}(x,y)}{f_Y(y)}\\ \\ \gg \gg \frac{1}{y}{e}^{-x/y}{e}^{-y}\cdot (e^{-y}{)}^{-1}\gg \gg \frac{1}{y}{e}^{-x/y}\end{array}$$

---

(3) Use $f_{X|Y}(x|y)$ to calculate expectation conditioned on the variable event:

$$\begin{array}{c}E[X\mid Y=y]\gg \gg {\int }_{-\infty }^{+\infty }x{f}_{X|Y}(x|y)dx\\ \\ \gg \gg {\int }_0^{\infty }\frac{x}{y}{e}^{-x/y}dx\gg \gg y\end{array}$$ Link to original

Theory - extra (non-examinable)

Theory 2

Expectation conditioned by a random variable

Suppose $X$ and $Y$ are any two random variables. The expectation of $X$ conditioned on $Y$ is a random variable giving the typical value of $X$ on the assumption that $Y$ has value determined by an outcome of the experiment.

$$E[X\mid Y]=g(Y)\text{where}g(y)=E[X\mid Y=y]$$

In other words, start by defining a function $g(y)$:

$$\begin{array}{rl}g:\mathbb{R}& \to \mathbb{R}\\ y& \mapsto E[X\mid Y=y]\end{array}$$

Now $E[X\mid Y]$ is defined as the composite random variable $g(Y)$.

Considered as a random variable, $E[X\mid Y]$ takes an outcome $s\in S$, computes $Y(s)$, sets $y=Y(s)$, then returns the expectation of $X$ conditioned on $Y=y$.

Notice that $X$ is not evaluated at $s$, only $Y$ is.

Because the value of $E[X\mid Y]$ depends only on $Y(s)$, and not on any additional information about $s$, it is common to represent a conditional expectation $E[X\mid Y]$ using only the function $g$.

---

Iterated Expectation

$$E[E[X\mid Y]]=E[X]$$

Proof of Iterated Expectation, discrete case

$$\begin{array}{rl}E[E[X\mid Y]]& =\sum _{\ell }E[X\mid Y=\ell ]P_Y(\ell )\\ & \\ & =\sum _{\ell }\sum _{k}k{P}_{X|Y}(k|\ell )P_Y(\ell )\\ & \\ & =\sum _{k}k\sum _{\ell }{P}_{X,Y}(k,\ell )\\ & \\ & =\sum _{k}k{P}_X(k)=E[X]\end{array}$$

Link to original

Illustration

Example - Conditional Expectation Variable

Sum of random number of RVs

Let $N$ denote the number of customers that enter a store on a given day.

Let $X_i$ denote the amount spent by the $i^{th}$ customer.

Assume that $E[N]=50$ and $E[X_i]=$ $8 for each $i$.

What is the expected total spend of all customers in a day?

Solution

A formula for the total spend is $X={\sum }_{i=1}^N{X}_i$.

By Iterated Expectation, we know $E[X]=E[E[X\mid N]]$.

Now compute $E[X\mid N]$ as a function of $N$:

$$\begin{array}{rl}E[X\mid N=n]& \gg \gg E\left[\left(\sum _{i=1}^N{X}_i\right)\mid N=n\right]\\ & \\ & \gg \gg E\left[\left(\sum _{i=1}^n{X}_i\right)\mid N=n\right]\\ & \\ & \gg \gg \sum _{i=1}^{n}E[X_i\mid N=n]\\ & \\ & \gg \gg \sum _{i=1}^{n}E[X_i]\gg \gg 8n\end{array}$$

Therefore $g(n)=8n$ and $g(N)=8N$ and $E[X\mid N]=8N$.

Then by Iterated Expectation, $E[X]=E[8N]=8E[N]=$ $400.

Link to original