Chapter 05 - Joint Distributions

Joint distributions

Theory

Theory 1

Joint distributions describe the probabilities of events associated with multiple random variables simultaneously.

In this course we consider only two variables at a time, typically called $X$ and $Y$. It is easy to extend this theory to vectors of $n$ random variables.

Joint PMF and joint PDF

Discrete joint PMF:

$$P_{X,Y}(x,y)=P[X=x,Y=y]$$

Continuous joint PDF:

$$f_{X,Y}(x,y)=\text{density at}(x,y)$$

Probabilities of events: Discrete case
If $B\subset {\mathbb{R}}^2$ is a set of points in the plane, then an event $\mathcal{B}$ is formed by the set of all outcomes $s$ mapped by $X$ and $Y$ to points in $B$:

$$\mathcal{B}=\{s\in S|(X(s),Y(s))\in B\}$$

The probabilities of such events can be measured using the joint PMF:

$$P[(X,Y)\in B]=P[\mathcal{B}]=\sum _{(x,y)\in B}{P}_{X,Y}(x,y)$$

Probabilities of events: Continuous case
Let $\mathcal{V}=[a,b]\times [c,d]\subset {\mathbb{R}}^2$ be the rectangular region defined by $(x,y)\in {\mathbb{R}}^2$ such that $a\le x\le b$ and $c\le y\le d$. Then:

$$P[(x,y)\in \mathcal{V}]=P[a\le X\le b,c\le Y\le d]={\int }_c^d{\int }_a^b{f}_{X,Y}(x,y)dxdy$$

For more general regions $\mathcal{V}\subset {\mathbb{R}}^2$:

$$P[(X,Y)\in \mathcal{V}]={\iint }_{\mathcal{V}}{f}_{X,Y}(x,y)dA$$

---

The existence of a variable $Y$ does not change the theory for a variable $X$ considered by itself.

However, it is possible to relate the theory for $X$ to the theory for $(X,Y)$, in various ways.

The simplest relationship is the marginal distribution for $X$, which is merely the distribution of $X$ itself, considered as a single random variable, but in a context where it is derived from the joint distribution for $(X,Y)$.

Marginal PMF, marginal PDF

Marginal distributions are obtained from joint distributions by summing the probabilities over all possibilities of the other variable.

Discrete marginal PMF:

$$\begin{array}{rl}{P}_X(x)& =\sum _y{P}_{X,Y}(x,y)\\ & \\ P_Y(y)& =\sum _x{P}_{X,Y}(x,y)\end{array}$$

Continuous marginal PMF:

$$\begin{array}{rl}{f}_X(x)& ={\int }_{-\infty }^{+\infty }{f}_{X,Y}(x,y)dy\\ & \\ f_Y(y)& ={\int }_{-\infty }^{+\infty }{f}_{X,Y}(x,y)dx\end{array}$$

---

Infinitesimal method

Suppose $X$ has density $f_X(x)$ that is continuous at $x_0$. Then for infinitesimal $dx$:

$$P[x_0<X\le x_0+dx]=f_X(x)dx$$

Suppose $X$ and $Y$ have joint density $f_{X,Y}(x,y)$ that is continuous at $(x_0,y_0)$. Then for infinitesimal $dx,dy$:

$$P[x_0<X\le x_0+dx,y_0<Y\le y_0+dy]=f_{X,Y}(x_0,y_0)dxdy$$

Joint densities depend on coordinates

The density $f_{X,Y}(x,y)$ in these integration formulas depends on the way $X$ and $Y$ act as Cartesian coordinates and determine differential areas $dxdy$ as little rectangles.

To find a density $f_{R,\Theta }(r,\theta )$ in polar coordinates, for example, it is not enough to solve for $x(r,\theta )$ and $y(r,\theta )$ and plug into $f_{X,Y}$!

Instead, we must consider the differential area $dxdy$ vs. $drd\theta$. We find that $dxdy=rdrd\theta$.

As an example, the density of the uniform distribution on the unit disk is $f_{R,\Theta }=\frac{r}{\pi }$, which is not constant as a function of $r$ and $\theta$.

Extra - Joint densities may not exist

It is not always possible to form a joint PDF $f_{X,Y}$ from any two continuous RVs $X$ and $Y$.

For example, if $X=Y$, then $(X,Y)$ cannot have a joint PDF, since $P[X=Y]=1$ but the integral over the region $X=Y$ will always be 0. (The area of a line is zero.)

Link to original

Illustration

Example - Smaller and bigger rolls

Joint and marginal PMFs - Smaller and bigger roll

Roll two dice, and let $X$ indicate the smaller of the numbers rolled, and let $Y$ indicate the bigger number.

Make a chart showing the PMF. Compute the marginal probabilities, and write them in the margins of the chart.

Solution

Link to original

Exercise - Event probabilities by reading PMF table

Event probabilities by reading PMF table

Here is a joint PMF table:

$$\begin{array}{ccccc}{P}_{Q,G}(q,g)& g=0& g=1& g=2& g=3\\ q=0& 0.06& 0.18& 0.24& 0.12\\ q=1& 0.04& 0.12& 0.16& 0.08\end{array}$$

Using the table, compute the following event probabilities:

(a) $P[Q=0]$ (b) $P[Q=G]$ (c) $P[G>1]$ (d) $P[G>Q]$

Link to original

Exercise - Joint and marginal PMFs - Coin flipping

Joint and marginal PMFs - Coin flipping

Flip a fair coin four times. Let $X$ measure the number of heads in the first two flips, and let $Y$ measure the total number of heads.

Make a chart showing the PMF. Compute the marginal probabilities, and write them in the margins of the chart.

Link to original

Example - Marginal and event probability from joint density

Marginal and event probability from joint density

Suppose the joint density of $X$ and $Y$ is given by:

$$f_{X,Y}(x,y)=\{\begin{array}{ll}2x{e}^{x^2-y}& y>x^2,x\in [0,1]\\ 0& \text{otherwise}\end{array}$$

Find (a) $f_Y(y)$ and (b) $P[Y<3X^2]$.

Solution

(a)

When $y\in [0,1]$, the $x$ range is $0$ to $\sqrt{y}$:

$$\begin{array}{c}{f}_Y(y)={\int }_{-\infty }^{+\infty }{f}_{X,Y}(x,y)dx\\ \\ \gg \gg {\int }_0^{\sqrt{y}}2x{e}^{x^2-y}dx\gg \gg 1-e^{-y}\text{for }0\le y\le 1\end{array}$$

When $y>1$, the $x$ range is $0$ to $1$:

$$\begin{array}{c}{f}_Y(y)={\int }_{-\infty }^{+\infty }{f}_{X,Y}(x,y)dx\\ \\ \gg \gg {\int }_0^{1}2x{e}^{x^2-y}dx\gg \gg e^{1-y}-e^{-y}\text{for }y>1\end{array}$$

Therefore:

$$f_Y(y)=\{\begin{array}{ll}0& y<0\\ 1-e^{-y}& 0\le y<1\\ (e-1)e^{-y}& 1\le y\end{array}$$

---

(b)

Find probability of the event $Y<3X^2$:

$$\begin{array}{c}P[Y<3X^2]={\int }_0^1{\int }_{x^2}^{3x^2}2x{e}^{x^2-y}dydx\\ \\ \gg \gg {\int }_0^{1}2x{e}^{x^2}\left(e^{-x^2}-e^{-3x^2}\right)dx\\ \\ \gg \gg \frac{1}{2}\left(1+e^{-2}\right)\end{array}$$ Link to original

Exercise - Marginals from joint density

Marginals from joint density

The joint PDF for $X$ and $Y$ is given by:

$$f_{X,Y}(x,y)=\{\begin{array}{ll}6\left(x+y^2\right)/5& 0\le x,y\le 1\\ 0& \text{ otherwise }\end{array}$$

Find $f_X(x)$ and $f_Y(y)$.

Link to original

Exercise - Event probability from joint density

Event probability from joint density

The joint PDF for $X$ and $Y$ is given by:

$$f_{X,Y}(x,y)=\{\begin{array}{ll}2e^{-x}{e}^{-2y}& x,y>0\\ 0& \text{else}\end{array}$$

Compute $P[X<Y]$.

Link to original

Theory

Theory 2

Joint CDF

The joint CDF of $X$ and $Y$ is defined by:

$$F_{X,Y}(x,y)=P[X\le x,Y\le y]$$

We can relate the joint CDF to the joint PDF using integration:

$$F_{X,Y}(x,y)={\int }_{-\infty }^y{\int }_{-\infty }^x{f}_{X,Y}(s,t)dsdt$$

Conversely, if $X$ and $Y$ have a continuous joint PDF $f_{X,Y}(x,y)$ that is also differentiable, we can obtain the PDF from the CDF using partial derivatives:

$$f_{X,Y}(x,y)=\frac{{\partial }^2}{\partial x\partial y}{F}_{X,Y}(x,y)$$

---

There is also a marginal CDF that is computed using a limit:

$$F_X(x)=\underset{y\to +\infty }{\lim }{F}_{X,Y}(x,y)$$

This could also be written, somewhat abusing notation, as $F_X(x)=F_{X,Y}(x,+\infty )$.

Link to original

Illustration

Exercise - Properties of joint CDFs

Properties of joint CDFs

(a) Show with a drawing that if both $x<x^{\prime }$ and $y<y^{\prime }$, we know:

$$F_{X,Y}(x,y)\le F_{X,Y}(x^{\prime },y^{\prime })$$

(b) Explain why:

• $F_X(x)=F_{X,Y}(x,\infty )$
• $F_Y(y)=F_{X,Y}(\infty ,y)$

(c) Explain why:

• $F_{X,Y}(x,-\infty )=0$
• $F_{X,Y}(-\infty ,y)=0$

Link to original

Independent random variables

Theory

Theory 1

Independent random variables

Random variables $X,Y$ are independent when they satisfy the product rule for all valid subsets $B_1,B_2\subset \mathbb{R}$:

$$P[X\in B_1,Y\in B_2]=P[X\in B_1]\cdot P[Y\in B_2]$$

Since $\{X\in B_1,Y\in B_2\}=\{X\in B_1\}\cap \{Y\in B_2\}$, this definition is equivalent to independence of all events constructible using the variables $X$ and $Y$.

For discrete random variables, it is enough to check independence for simple events of type $\{X=k\}$ and $\{Y=\ell \}$ for $k$ and $\ell$ any possible values of $X$ and $Y$.

---

The independence criterion for random variables can be cast entirely in terms of their distributions and written using the PMFs or PDFs.

Independence using PMF and PDF

Discrete case:

$$P_{X,Y}(x,y)=P_X(x)\cdot P_Y(y)$$

Continuous case:

$$f_{X,Y}(x,y)=f_X(x)\cdot f_Y(y)$$

Independence via joint CDF

Random variables $X$ and $Y$ are independent when their CDFs obey the product rule:

$$F_{X,Y}(x,y)=F_X(x)\cdot F_Y(y)$$

Link to original

Illustration

Example - Meeting in the park

Event probability - Meeting in the park

A man and a woman arrange to meet in the park between 12:00 and 1:00pm. They both arrive at a random time with uniform distribution over that hour, and do not coordinate with each other.

Find the probability that the first person to arrive has to wait longer than 15 minutes for the second person to arrive.

Solution

Let $X$ denote the time the man arrives. Use minutes starting from 12:00, so $X\in (0,60)$. Let $Y$ denote the time the woman arrives, using the same interval.

The probability we seek is:

$$P[X+15<Y]+P[Y+15<X]$$

Because $X$ and $Y$ are symmetrical in probability, these terms have the same value, so we just double the first one for our answer.

Since the arrivals are independent of each other, we have $f_{X,Y}=f_X\cdot f_Y$.

Since each arrival time is uniform over the interval, we have:

$$f_X(x)=\{\begin{array}{ll}1/60& x\in (0,60)\\ 0& \text{otherwise}\end{array}{f}_Y(y)=\{\begin{array}{ll}1/60& y\in (0,60)\\ 0& \text{otherwise}\end{array}$$

Therefore the joint density is $f_{X,Y}={\left(\frac{1}{60}\right)}^2$. Calculate:

$$\begin{array}{c}2P[X+15<Y]\gg \gg 2{\iint }_{x+15<y}{f}_{X,Y}(x,y)dxdy\\ \\ \gg \gg 2{\iint }_{x+15<y}{f}_X(x)f_Y(y)dxdy\gg \gg 2{\int }_{15}^{60}{\int }_0^{y-15}{\left(\frac{1}{60}\right)}^{2}dxdy\\ \\ \gg \gg \frac{2}{(60{)}^2}{\int }_{15}^{60}y-15dy\gg \gg \frac{9}{16}\end{array}$$ Link to original

Example - Uniform disk: Cartesian vs. polar

Uniform disk: Cartesian vs. polar

Suppose that a point is chosen uniformly at random on the unit disk.

(a) Let $X$ and $Y$ be the Cartesian coordinates of the chosen point. Are $X$ and $Y$ independent?

(b) Let $R$ and $\Theta$ give the polar coordinates of the chosen point. Are $R$ and $\Theta$ independent?

Solution

(a)

Write $f_{X,Y}$ for the joint distribution of $X$ and $Y$. We have:

$$f_{X,Y}=\{\begin{array}{ll}1/\pi & x^2+y^2\le 1\\ 0& \text{otherwise}\end{array}$$

Then computing $f_X(x)$, we obtain:

$$\begin{array}{c}{\int }_{-\sqrt{1-x^2}}^{+\sqrt{1-x^2}}\frac{1}{\pi }dy\gg \gg \frac{2}{\pi }\sqrt{1-x^2}\\ \\ \gg \gg f_X(x)=\{\begin{array}{ll}\frac{2}{\pi }\sqrt{1-x^2}& x\in [-1,1]\\ 0& \text{otherwise}\end{array}\end{array}$$

By similar reasoning, $f_Y(y)=\frac{2}{\pi }\sqrt{1-y^2}$ for $y\in [-1,1]$.

The product $f_X(x)f_Y(y)$ is not equal to $f_{X,Y}(x,y)$, so $X$ and $Y$ are not independent. Information about the value of $X$ does provide constraints on the possible values of $Y$, so this result makes sense.

---

(b)

To find the marginals $f_R(r)$ and $f_{\Theta }(\theta )$, the standard method is to integrate the density $f_{R,\Theta }$ in the opposite variables.

$f_{R,\Theta }(r,\theta )$ varies!

The probability density $f_{R,\Theta }(r,\theta )$ is not constant!
The area of a differential sector $drd\theta$ depends on $r$.

We can take two approaches to finding the density $f_{R,\Theta }$:

(i) Area of a differential sector divided by total area:

$$\frac{rdrd\theta }{\pi }\gg \gg \frac{r}{\pi }drd\theta$$

So the density is $f_{R,\Theta }=\frac{r}{\pi }$.

(ii) Via the CDF:

Joint PDF from joint CDF: uniform distribution in polar

The region ‘below’ a given point $(r,\theta )$, in polar coordinates, is a sector with area $\frac{\theta }{2\pi }\cdot \pi r^2$. The factor $\frac{\theta }{2\pi }$ is a percentage of the circle with area $\pi r^2$.

The density is a constant $\frac{1}{\pi }$ across the disk, so the CDF at $(r,\theta )$ is this same area times $\frac{1}{\pi }$. Thus:

$$F_{R,\Theta }=\frac{\theta r^2}{2\pi }$$

Then in polar coordinates the density is given by taking partial derivatives:

$$f_{R,\Theta }(r,\theta )=\frac{{\partial }^2}{\partial r\partial \theta }\left(\frac{1}{2\pi }\theta r^2\right)\gg \gg \frac{r}{\pi }$$

Once we have $f_{R,\Theta }$, integrate to get the marginals:

$$\begin{array}{rl}{f}_R(r)& ={\int }_{\theta =0}^{2\pi }{f}_{R,\Theta }d\theta \gg \gg {\int }_0^{2\pi }\frac{r}{\pi }d\theta \gg \gg 2r\\ & \\ f_{\Theta }(\theta )& ={\int }_{r=0}^1{f}_{R,\Theta }dr\gg \gg {\int }_0^1\frac{r}{\pi }dr\gg \gg \frac{1}{2\pi }\end{array}$$

Check independence:

$$f_{R,\Theta }=\frac{r}{\pi }\check{=}(2r)\left(\frac{1}{2\pi }\right)=f_R\cdot f_{\Theta }$$

---

In this problem it is feasible to find the marginals directly, without computing the new density, only using some geometric reasoning.

Extra - Infinitesimal method for the marginals

The probability $P[R\in (r,r+dr)]$ is the area (over $\pi$) of a thickened circle with radius $r$ and thickness $dr$. The circumference of a circle at radius $r$ is $2\pi r$. So the area of the thickened circle is $2\pi rdr$. So the probability is $2rdr$. This tells us that the marginal probability density is $P_R(r)=2r$.

The probability $P[\Theta \in (\theta ,\theta +d\theta )]$ is the area (over $\pi$) of a thin sector with radius 1 and angle $d\theta$. This area is $\frac{1}{2}{1}^{2}d\theta$. So the probability is $\frac{1}{2\pi }d\theta$. This tells us that the marginal probability density is $P_{\Theta }(\theta )=\frac{1}{2\pi }$.

These results agree with those above from differentiating the CDF.

Link to original

Functions on two random variables

Theory

Theory 1

PMF of (any) function of two discrete variables

Suppose $W=g(X,Y)$ and $X,Y$ are discrete RVs.

The PMF of $W$:

$$P_W(w)=\sum _{\stackrel{(x,y)\text{ s.t.}}{g(x,y)=w}}{P}_{X,Y}(x,y)$$

CDF of (continuous) function of two continuous variables

Suppose $W=g(X,Y)$ and $X,Y$ are continuous RVs, and $g$ is a continuous function.

The CDF of $W$:

$$F_W(w)=P[W\le w]=\underset{g(x,y)\le w}{\iint }{f}_{X,Y}(x,y)dxdy$$

If desired, one can then compute the PDF of $W$ by differentiating the continuous CDF:

$$f_W(w)=\frac{d}{dw}{F}_W(w)$$Link to original

Illustration

Exercise - PMF of $X{Y}^2$ from chart

PMF of XY squared from chart

Suppose the joint PMF of $X$ and $Y$ is given by this chart:

$Y\downarrow X\to$	$1$	$2$
$-1$	0.2	0.2
$0$	0.35	0.1
$1$	0.05	0.1

Define $W=X{Y}^2$.

(a) Find the PMF $P_W(w)$.

(b) Find the expectation $E[W]$.

Link to original

Example - Max and Min from joint PDF

Max and Min from joint PDF

Suppose the joint PDF of $X$ and $Y$ is given by:

$$f_{X,Y}(x,y)=\{\begin{array}{ll}\frac{3}{2}(x^2+y^2)& x,y\in [0,1]\\ 0& \text{otherwise}\end{array}$$

Find the PDF of (a) $W=\mathrm{Max}(X,Y)$, and of (b) $W=\mathrm{Min}(X,Y)$.

Solution

(a)

(1) Compute CDF of $W$:

Convert to event form:

$$\begin{array}{c}{F}_W(w)=P[\mathrm{Max}(X,Y)\le w]\\ \\ \gg \gg P[X\le w\text{and}Y\le w]\end{array}$$

Integrate PDF over the region, assuming $w\in [0,1]$:

$$\begin{array}{c}{\int }_{-\infty }^w{\int }_{-\infty }^w{f}_{X,Y}(x,y)dxdy\\ \\ \gg \gg {\int }_0^w{\int }_0^w\frac{3}{2}(x^2+y^2)dxdy\gg \gg w^4\end{array}$$

---

(2) Differentiate to find $f_W(w)$:

$f_W=\frac{d}{dw}{F}_W(w)$:

$$f_W(w)=\{\begin{array}{ll}4w^3& w\in [0,1]\\ 0& \text{otherwise}\end{array}$$

---

(b)

(1) Compute CDF of $W$:

Convert to event form:

$$\begin{array}{c}{F}_W(w)=P[\mathrm{Min}(X,Y)\le w]\\ \\ \gg \gg 1-P[\mathrm{Min}(X,Y)>w]\\ \\ \gg \gg 1-P[X>w\text{and}Y>w]\end{array}$$

Integrate PDF over the region:

$$\begin{array}{c}P[X>w\text{and}Y>w]\gg \gg {\int }_w^1{\int }_w^1\frac{3}{2}(x^2+y^2)dxdy\\ \\ \gg \gg w^4-w^3-w+1\end{array}$$

Therefore:

$$F_W(w)=-w^4+w^3+w$$

---

(2) Differentiate to find $f_W(w)$:

$f_W=\frac{d}{dw}{F}_W(w)$:

$$f_W(w)=\{\begin{array}{ll}-4w^3+3w^2+1& w\in [0,1]\\ 0& \text{otherwise}\end{array}$$ Link to original

Example - PDF of a sum

PDF of sums practice

Suppose $X$ is an RV with density:

$$f_X=\{\begin{array}{ll}2x& x\in [0,1]\\ 0& \text{otherwise}\end{array}$$

Suppose $Y$ is uniform on $[0,1]$ and independent of $X$.

Find the PDF of $X+Y$. Sketch the graph of this PDF.

Solution

(1) Write the CDF of $W=X+Y$ as a double integral:

$$F_W(w)=P[X+Y\le w]={\iint }_{x+y\le w}{f}_{X,Y}dxdy$$

The joint density on the unit square $x\in [0,1],y\in [0,1]$ is:

$$f_{X,Y}\gg \gg f_X\cdot f_Y\gg \gg 2x\cdot 1\gg \gg 2x$$

There is positive density in the region $x+y\le w$ only for $x\le w$ (otherwise $y<0$).

• When $w\in [0,1]$, there is positive density in the region (only) when $y\le w-x$.
• When $w\in [1,2]$, there is positive density in the region whenever $y\le 1$.

---

(2) Evaluate $F_W(w)$ for $w\in [0,1]$:

Here $x\in [0,w]$ and $w-x\le 1$, so $y\in [0,w-x]$.

$$\begin{array}{c}{F}_W(w)={\int }_0^w{\int }_0^{w-x}2xdydx\\ \\ \gg \gg {\int }_0^{w}2x(w-x)dx\gg \gg {\left[w{x}^2-\frac{2x^3}{3}\right]}_0^w\\ \\ \gg \gg w^3-\frac{2w^3}{3}\gg \gg \frac{w^3}{3}\end{array}$$

Differentiate:

$$f_W(w)=\frac{d}{dw}\frac{w^3}{3}\gg \gg w^2$$

---

(3) Evaluate $F_W(w)$ for $w\in [1,2]$:

Now $x$ ranges over $[0,1]$. Split by whether the $y$-bound is $1$ or $w-x$:

• $x\in [0,w-1]$: $w-x\ge 1$, so $y\in [0,1]$
• $x\in [w-1,1]$: $w-x<1$, so $y\in [0,w-x]$

$$\begin{array}{c}{F}_W(w)={\int }_0^{w-1}{\int }_0^{1}2xdydx+{\int }_{w-1}^1{\int }_0^{w-x}2xdydx\\ \\ \gg \gg {\int }_0^{w-1}2xdx+{\int }_{w-1}^{1}2x(w-x)dx\\ \\ \gg \gg (w-1{)}^2+{\left[w{x}^2-\frac{2x^3}{3}\right]}_{w-1}^1\\ \\ \gg \gg (w-1{)}^2+(w-\frac{2}{3})-\left(w(w-1{)}^2-\frac{2(w-1{)}^3}{3}\right)\\ \\ \gg \gg -\frac{1}{3}{x}^3+w^2-\frac{1}{3}\end{array}$$

---

(4) Differentiate for the final PDF:

$$f_W(w)=\frac{d}{dw}\left(-\frac{1}{3}{x}^3+w^2-\frac{1}{3}\right)\gg \gg -w^2+2w$$

Therefore:

$$f_{X+Y}(w)=\{\begin{array}{ll}{w}^2& w\in [0,1]\\ & \\ -w^2+2w& w\in [1,2]\\ & \\ 0& \text{otherwise}\end{array}$$ Link to original

Extra - Convolution

Convolution

Extra Example - PDF of a quotient

PDF of a quotient

Suppose the joint PDF of $X$ and $Y$ is given by:

$$f_{X,Y}(x,y)=\{\begin{array}{ll}\lambda \mu e^{-(\lambda x+\mu y)}& x,y\ge 0\\ 0& \text{otherwise}\end{array}$$

Find the PDF of $W=g(X,Y)$ for $g(X,Y)=Y/X$.

Solution

(1) Find the CDF using logic:

$$\begin{array}{c}{F}_W(w)=P[Y/X\le w]\\ \\ \gg \gg P[Y\le wX]\end{array}$$

Integrate over this region:

$$\begin{array}{rl}P[Y\le wX]& ={\int }_0^{\infty }{\int }_0^{wx}{f}_{X,Y}(x,y)dydx\\ & \\ & \gg \gg {\int }_0^{\infty }\lambda e^{-\lambda x}{\int }_0^{wx}\mu e^{-\mu y}dydx\\ & \\ & \gg \gg {\int }_0^{\infty }\lambda e^{-\lambda x}\left(-e^{-\mu wx}+1\right)dx\\ & \\ & \gg \gg 1-\frac{\lambda }{\lambda +\mu w}\end{array}$$

---

(2) Differentiate to find PDF:

Compute $\frac{d}{dw}{F}_W(w)$:

$$f_W(w)=\{\begin{array}{ll}\frac{\lambda \mu }{(\lambda +\mu w{)}^2}& w\ge 0\\ & \\ 0& \text{otherwise}\end{array}$$Link to original

Theory

Theory 6

Recall that in a Poisson process:

• $X\sim \mathrm{Exp}(\lambda )$ measures continuous wait time until one arrival
• $X\sim \mathrm{Erlang}(\ell ,\lambda )$ measures continuous wait time until ${\ell }^{\text{th}}$ arrival

Since the wait times between arrivals are independent, we expect that the sum of exponential distributions is an Erlang distribution. This is true!

Erlang sum rule

Specify a given Poisson process with arrival rate $\lambda$. Suppose that:

• $X\sim \mathrm{Erlang}(r,\lambda )$ for any $r=1,2,3,\dots$
• $Y\sim \mathrm{Erlang}(s,\lambda )$ or any $s=1,2,3,\dots$
• $X$ and $Y$ are independent

Then:

$$X+Y\sim \mathrm{Erlang}(r+s,\lambda )$$

Exp plus Exp is Erlang

Recall that $\mathrm{Erlang}(1,\lambda )\sim \mathrm{Exp}(\lambda )$.

So we could say: $$\text{“}\mathrm{Exp}(\lambda )+\mathrm{Exp}(\lambda )\sim \mathrm{Erlang}(2,\lambda )\text{”}$$

And:

$$\text{“}\mathrm{Exp}(\lambda )+\mathrm{Erlang}(\ell ,\lambda )\sim \mathrm{Erlang}(\ell +1,\lambda )\text{”}$$

Link to original

Illustration

Example - Exp plus Exp equals Erlang

Exp plus Exp equals Erlang - Without Convolution

Let us verify this formula by direct calculation:

$$\text{“}\mathrm{Exp}(\lambda )+\mathrm{Exp}(\lambda )\sim \mathrm{Erlang}(2,\lambda )\text{”}$$

Solution

Let $X,Y\sim \mathrm{Exp}(\lambda )$ be independent RVs, and let $W=X+Y$.

Therefore:

$$f_{X,Y}(x,y)=\lambda e^{-\lambda x}\cdot \lambda e^{-\lambda y}={\lambda }^2{e}^{-\lambda (x+y)}x\ge 0,y\ge 0$$

---

(1) Write the CDF as a double integral over the region $x\ge 0,y\ge 0,x+y\le w$:

For $w\ge 0$, the region is $x\in [0,w]$, $y\in [0,w-x]$.

$$F_W(w)={\int }_0^w{\int }_0^{w-x}{\lambda }^2{e}^{-\lambda (x+y)}dydx$$

---

(2) Evaluate the inner integral:

$$\begin{array}{c}{\int }_0^{w-x}{\lambda }^2{e}^{-\lambda x}{e}^{-\lambda y}dy\gg \gg {\lambda }^2{e}^{-\lambda x}{\left[-\frac{1}{\lambda }{e}^{-\lambda y}\right]}_0^{w-x}\\ \\ \gg \gg \lambda e^{-\lambda x}(1-e^{-\lambda (w-x)})\\ \\ \gg \gg \lambda (e^{-\lambda x}-e^{-\lambda w})\end{array}$$

---

(3) Evaluate the outer integral:

$$\begin{array}{c}{F}_W(w)=\lambda {\int }_0^w(e^{-\lambda x}-e^{-\lambda w})dx\\ \\ \gg \gg \lambda {\left[-\frac{1}{\lambda }{e}^{-\lambda x}-x{e}^{-\lambda w}\right]}_0^w\\ \\ \gg \gg \lambda \left[\left(-\frac{1}{\lambda }{e}^{-\lambda w}-w{e}^{-\lambda w}\right)-\left(-\frac{1}{\lambda }\right)\right]\\ \\ \gg \gg 1-e^{-\lambda w}-\lambda w{e}^{-\lambda w}\end{array}$$

---

(4) Differentiate for the PDF:

$$\begin{array}{c}{f}_W(w)=\frac{d}{dw}(1-e^{-\lambda w}-\lambda w{e}^{-\lambda w})\\ \\ \gg \gg \lambda e^{-\lambda w}-\lambda e^{-\lambda w}+{\lambda }^{2}w{e}^{-\lambda w}\\ \\ \gg \gg {\lambda }^{2}w{e}^{-\lambda w}\end{array}$$

This is the $\mathrm{Erlang}(2,\lambda )$ density function:

$${\frac{{\lambda }^{\ell }}{(\ell -1)!}{t}^{\ell -1}{e}^{-\lambda t}|}_{\ell =2}$$ Link to original

Exercise - Erlang induction step

Erlang induction step

Derive the formula:

$$\text{“}\mathrm{Exp}(\lambda )+\mathrm{Erlang}(\ell ,\lambda )\sim \mathrm{Erlang}(\ell +1,\lambda )\text{”}$$

Observation: By repeatedly applying the above formula, we see that:

$$\text{“}\stackrel{\ell \text{ terms}}{\overbrace{\mathrm{Exp}(\lambda )+\cdots +\mathrm{Exp}(\lambda )}}\sim \mathrm{Erlang}(\ell ,\lambda )\text{”}$$Link to original