Chapter 08 - Testing and Estimation

Significance testing

Theory - Significance testing

Theory 1 - Significance testing

Significance test

Ingredients of a significance test (unary hypothesis test):

• $H_0$ — Null hypothesis event
  • Identify a Claim
  • Then: $H_0$ is background assumption (supposing Claim isn’t known)
  • Goal is to invalidate $H_0$ in favor of Claim
• $R$ — Rejection Region event (decision rule)
  • $R$ is written in terms of decision statistic $X$ and significance level $\alpha$
  • $R$ is unlikely assuming $H_0$. $R$ is more likely if Claim
  • “If $X$ falls in $R$, this Test rejects $H_0$.”
• $P[R\mid H_0]$ — Able to compute this
  • Usually: inferred from $f_{X|H_0}$ or $P_{X|H_0}$
  • Adjust $R$ to achieve $P[R\mid H_0]=\alpha$

Significance level

Suppose we are given a null hypothesis $H_0$ and a rejection region $R$.

The significance level of $R$ is:

$$\begin{array}{rl}\alpha & =P[R\mid H_0]\\ & \\ & =P[\text{reject }{H}_0\mid H_0\text{ is true}]\end{array}$$

Sometimes the condition is dropped and we write $\alpha =P[R]$, e.g. when a background model without assuming $H_0$ is not known.

Null hypothesis implies a distribution

Usually $S$ is unspecified, yet $H_0$ determines a known distribution.

In this case $H_0$ will not take the form of an event in a sample space, $H_0\subset S$.

At a minimum, $H_0$ must determine $P[R\mid H_0]$.

We do NOT need these details:

• Background sample space $S$
• Non-conditional distribution (full model): $f_X$ or $P_X$
• Complement conditionals: $f_{X|H_0^c}$ or $P_{X|H_0^c}$

---

In basic statistical inference theory, there are two kinds of error.

• Type I error concludes with rejecting $H_0$ when $H_0$ is true.
• Type II error concludes with maintaining $H_0$ when $H_0$ is false.

Type I error is usually a bigger problem. We want to consider $H_0$ as “innocent until proven guilty.”

	$H_0$ is true	$H_0$ is false
Maintain null hypothesis	Made right call	Wrong acceptance $Type\; II\; Error$
Reject null hypothesis	Wrong rejection $Type\; I\; Error$	Made right call

---

To design a significance test at $\alpha$, we must identify $H_0$, and specify $R$ with the property that $P[R\mid H_0]=\alpha$.

When $R$ is written using a variable $X$, we must choose between:

• One-tail rejection region: $x$ with $R(x)\le r$ or $x$ with $R(x)\ge r$
• Two-tail rejection region: $x$ with $|R(x)-\mu |\ge c$

Link to original

Illustration

Example - One-tail test: Weighted die

One-tail test: Weighted die

Your friend gives you a single regular die, and say she is worried that it has been weighted to prefer the outcome of 2. She wants you to test it.

Design a significance test for the data of 20 rolls of the die to determine whether the die is weighted. Use significance level $\alpha =0.05$.

Solution

Let $X$ count the number of 2s that come up.

The Claim: “the die is weighted to prefer 2”
The null hypothesis $H_0$: “the die is normal”

Assuming $H_0$ is true, then $X\sim \mathrm{Bin}(20,1/6)$, and therefore:

$$P_{X|H_0}(k)=\left(\genfrac{}{}{0ex}{}{20}{k}\right)(1/6{)}^k(5/6{)}^{20-k}$$

---

⚠️ Notice that “prefer 2” implies the claim is for more 2s than normal.

Therefore: Choose a one-tail rejection region.

Need $r$ such that:

$$\begin{array}{rl}P[X\ge r\mid H_0]& =0.05\\ & \\ ⟺P[X<r\mid H_0]& =0.95\end{array}$$

---

Solve for $r$ by computing conditional CDF values:

$k:$	0	1	2	3	4	5	6	7
$F_{X\mid H_0}(k):$	0.026	0.130	0.329	0.567	0.769	0.898	0.963	0.989

Therefore, choose $r=6$:

$P[X\ge 6\mid H_0]<0.04$, but $P[X\ge 5\mid H_0]>0.05$. Final answer:

$$R=\{x\mid x\ge 6\}$$ Link to original

Two-tail test: Circuit voltage

Two-tail test: Circuit voltage

A boosted AC circuit is supposed to maintain an average voltage of $130\mathrm{V}$ with a standard deviation of $2.1\mathrm{V}$. Nothing else is known about the voltage distribution.

Design a two-tail test incorporating the data of 40 independent measurements to determine if the expected value of the voltage is truly $130\mathrm{V}$. Use $\alpha =0.02$.

Solution

Use $M_{40}(V)$ as the decision statistic, i.e. the sample mean of 40 measurements of $V$.

The Claim to test: $E[V]\ne 130$

The null hypothesis $H_0$: $E[V]=130$

Rejection region:

$$|M_{40}-130|\ge c$$

where $c$ is chosen so that $P[|M_{40}-130|\ge c]=0.02$

---

Assuming $H_0$, we expect that:

$$E[M_{40}]=130,{\sigma }_{M_{40}}^2=\frac{2.1^2}{40}\approx 0.110$$

Recall Chebyshev’s inequality:

$$P[|M_{40}-130|\ge c]\le \frac{{\sigma }_{M_{40}}^2}{c^2}\approx \frac{0.110}{c^2}$$

---

Now solve:

$$\frac{0.110}{c^2}=0.02\gg \gg c\approx 2.348$$

Therefore the rejection region should be:

$$M_{40}<127.65\cup 132.35<M_{40}$$ Link to original

One-tail test with a Gaussian: Weight loss drug

One-tail test with a Gaussian: Weight loss drug

Assume that in the background population in a specific demographic, the distribution of a person’s weight $W$ satisfies $W\sim \mathcal{N}(190,24^2)$. Suppose that a pharmaceutical company has developed a weight-loss drug and plans to test it on a group of 64 individuals.

Design a test at the $\alpha =0.01$ significance level to determine whether the drug is effective.

Solution

Since the drug is tested on 64 individuals, we use the sample mean $M_{64}(W)$ as the decision statistic.

The Claim: “the drug is effective in reducing weight”

The null hypothesis $H_0$: “no effect: weights on the drug still follow $\mathcal{N}(190,24^2)$”

Assuming $H_0$ is true, then $W\sim \mathcal{N}(190,24^2)$.

⚠️ One-tail test because the drug is expected to reduce weight (unidirectional). Rejection region:

$$M_{64}(W)\le r$$

---

Calculate:

$${\sigma }_{M_{64}}^2\gg \gg \frac{24^2}{64}\gg \gg 9$$

⚠️ Standardized $M_{64}(W)$ is approximately normal!

(The standardization of $M_{64}(W)$ removes the effect of $\frac{1}{n}$. As if it’s the summation.)

So, standardize and apply CLT:

$$\begin{array}{c}\frac{M_{64}(W)-190}{\sqrt{9}}\sim \mathcal{N}(0,1),\\ \\ \gg \gg P[M_{64}(W)\le r]\approx P\left[Z\le \frac{r-190}{3}\right]=\Phi \left(\frac{r-190}{3}\right)\end{array}$$

---

Solve:

$$\begin{array}{c}P[M_{64}(W)\le r]=0.01\\ \\ \gg \gg \Phi \left(\frac{r-190}{3}\right)=0.01\\ \\ \gg \gg \Phi \left(\frac{190-r}{3}\right)=0.99\\ \\ \gg \gg \frac{190-r}{3}=2.33\\ \\ \gg \gg r=183.01\end{array}$$

Therefore, the rejection region:

$$M_{64}(W)\le 183.01$$Link to original

Binary hypothesis testing

Theory - Binary testing, MAP and ML

Theory 1 - Binary testing, MAP and ML

Binary hypothesis test

Ingredients of a binary hypothesis test:

• $H_0$ and $H_1$ — Complementary hypotheses
  • Maybe also know the prior probabilities $P[H_0]$ and $P[H_1]$
  • Goal: determine which case we are in, $H_0$ or $H_1$
• $A_0$ and $A_1$ — Complementary events of the Decision Rule
  • Directionality: given $H_0$, $A_0$ is likely; given $H_1$, $A_1$ is likely
  • Decision Rule: outcome $A_0$, accept $H_0$; outcome $A_1$, accept $H_1$
  • Usually: $A_i$ written in terms of decision statistic $X$ using a design
  • We cover three designs:
    • MAP and ML (minimize ‘error probability’)
    • MC (minimizes ‘error cost’)
  • Designs use $P_{X|H_0}$ and $P_{X|H_1}$ (or $f_{X|H_0}$, $f_{X|H_1}$) to construct $A_0$ and $A_1$

MAP design

Suppose we know:

• $P[H_0]$ and $P[H_1]$
  • Both prior probabilities
• $P_{X|H_0}(x)$ and $P_{X|H_1}(x)$ (or $f_{X|H_0}(x)$ and $f_{X|H_1}(x)$)
  • Both conditional distributions

The maximum a posteriori probability (MAP) design for a decision statistic $X$:

$$A_0=\text{set of x for which:}$$

Discrete case:

$$P_{X\mid H_0}(x)\cdot P[H_0]\ge P_{X\mid H_1}(x)\cdot P[H_1]$$

Continuous case:

$$f_{X\mid H_0}(x)\cdot P[H_0]\ge f_{X\mid H_1}(x)\cdot P[H_1]$$

And $A_1=\{x\in \mathbb{R}\mid x\notin A_0\}$.

The MAP design minimizes the total probability of error.

ML design

Suppose we don’t know the priors, we know only:

• $P_{X|H_0}(x)$ and $P_{X|H_1}(x)$ (or $f_{X|H_0}(x)$ and $f_{X|H_1}(x)$)
  • Both conditional distributions

The maximum likelihood (ML) design for $X$:

$$A_0=\text{set of x for which:}\begin{array}{rl}{P}_{X\mid H_0}(x)& \ge P_{X\mid H_1}(x)\text{(discrete)}\\ & \\ f_{X\mid H_0}(x)& \ge f_{X\mid H_1}(x)\text{(continuous)}\end{array}$$

ML is a simplified version of MAP. (Set $P[H_0]$ and $P[H_1]$ to $0.5$.)

---

The false alarm error rate is called $P_{FA}$. The miss error rate is called $P_{\text{Miss}}$.

$$\begin{array}{rl}{P}_{FA}& =P[A_1\mid H_0]\\ & \\ P_{\text{Miss}}& =P[A_0\mid H_1]\end{array}$$

Total probability of error:

$$P_{\text{ERR}}=P[A_1\mid H_0]\cdot P[H_0]+P[A_0\mid H_1]\cdot P[H_1]$$

Wrong meanings of $P_{FA}$

Suppose $A_1$ sets off a smoke alarm, and $H_0$ is ‘no fire’ and $H_1$ is ‘yes fire’.

Then $P_{FA}$ is the odds that we get an alarm assuming there is no fire.

This is not the odds of experiencing a false alarm (no context). That would be $P[A_1{H}_0]$.

This is not the odds of a given alarm being a false one. That would be $P[H_0\mid A_1]$.

Link to original

Illustration

Example - ML test: Smoke detector

ML test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $\mathcal{N}(0,3^2)$.

Design an ML test for the detector electronics to decide whether to activate the alarm.

What are the three error probabilities? (Type I, Type II, Total.)

Solution

First, establish the conditional distributions:

$$X\mid H_0\sim \mathcal{N}(0,3^2)X\mid H_1\sim \mathcal{N}(8,3^2)$$

Density functions:

$$f_{X|H_0}=\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}{f}_{X|H_1}=\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}$$

---

The ML condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2\\ \\ \gg \gg x^2\stackrel{?}{\le }(x-8{)}^2\\ \\ \gg \gg x\le 4\end{array}$$

---

Therefore, $A_0$ is $x\le 4$, while $A_1$ is $x>4$.

The decision rule is: activate alarm when $x>4$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1\mid H_0]\gg \gg P[X>4\mid H_0]\\ \\ \gg \gg 1-P\left[\frac{X-0}{3}\le \frac{4}{3}|H_0\right]\\ \\ \gg \gg 1-P[Z\le 1.3333]\gg \gg \approx 0.0912\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0\mid H_1]\gg \gg P[X\le 4\mid H_1]\\ \\ \gg \gg P\left[\frac{X-8}{3}\le \frac{4-8}{3}|H_1\right]\\ \\ \gg \gg P[Z\le -1.3333]\gg \gg \approx 0.0912\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.5+P_{\text{Miss}}\cdot 0.5\gg \gg \approx 0.0912$$ Link to original

Example - MAP test: Smoke detector

MAP test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $\mathcal{N}(0,3^2)$.

Suppose that the background chance of smoke is $5\%$. Design a MAP test for the alarm.

What are the three error probabilities? (Type I, Type II, Total.)

Solution

First, establish priors:

$$P[H_0]=0.95P[H_1]=0.05$$

The MAP condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\cdot 0.95\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot 0.05\\ \\ \gg \gg e^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot \frac{0.05}{0.95}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2+\ln \left(\frac{0.05}{0.95}\right)\\ \\ \gg \gg x^2\stackrel{?}{\le }(x-8{)}^2-18\ln \left(\frac{0.05}{0.95}\right)\\ \\ \gg \gg x\le 7.31\end{array}$$

---

Therefore, $A_0$ is $x\le 7.31$, while $A_1$ is $x>7.31$.

The decision rule is: activate alarm when $x>7.31$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1\mid H_0]\gg \gg P[X>7.31\mid H_0]\\ \\ \gg \gg 1-P[Z\le 2.4367]\gg \gg \approx 0.007411\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0\mid H_1]\gg \gg P[X\le 7.31\mid H_1]\\ \\ \gg \gg P[Z\le -0.23]\gg \gg \approx 0.4090\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.95+P_{\text{Miss}}\cdot 0.05\approx 0.02749$$ Link to original

Theory - MAP criterion proof

Theory 2 - MAP criterion proof

Explanation of MAP criterion - discrete case

First, we show that the MAP design selects for $A_0$ all those $x$ which render $H_0$ more likely than $H_1$. This will be used in the next step to show that MAP minimizes probability of error.

Observe this calculation:

$$\begin{array}{rl}P[H_i\mid X=x]& =P[X=x\mid H_i]\cdot \frac{P[H_i]}{P[X=x]}\\ & \\ & =P_{X|H_i}(x)\cdot \frac{P[H_i]}{P[X=x]}\end{array}\text{\hspace{1em}(Bayes’ Rule)(Conditional PMF)}$$

Recall the MAP criterion:

$$P_{X|H_0}(x)\cdot P[H_0]\ge P_{X|H_1}(x)\cdot P[H_1]$$

Divide both sides by $P[X=x]$ and apply the above Calculation in reverse:

$$\gg \gg P[H_0\mid X=x]\ge P[H_1\mid X=x]$$

This is what we sought to prove.

---

Next, we verify that the MAP design minimizes the total probability of error.

The total probability of error is:

$$P_{\mathrm{ERR}}=P[A_1\mid H_0]\cdot P[H_0]+P[A_0\mid H_1]\cdot P[H_1]$$

Expand this with summation notation (assuming the discrete case):

$$\gg \gg \sum _{x\in A_1}{P}_{X|H_0}(x)\cdot P[H_0]+\sum _{x\in A_0}{P}_{X|H_1}(x)\cdot P[H_1]$$

Now, how do we choose the set $A_0\subset \mathbb{R}$ (and thus $A_1=A_0^c$) in such a way that this sum is minimized?

Since all terms are positive, and any $x\in \mathbb{R}$ may be placed in $A_1$ or in $A_0$ freely and independently of all other choices, the total sum is minimized when we minimize the impact of placing each $x$.

So, for each $x$, we place it in $A_0$ if:

$$P_{X|H_0}(x)\cdot P[H_0]\ge P_{X|H_1}(x)\cdot P[H_1]$$

That is equivalent to the MAP criterion.

Link to original

Theory - MC design

Theory 3 - MC design

• Write $C_{10}$ for cost of false alarm, i.e. cost when $H_0$ is true but decided $H_1$.
  • Probability of incurring cost $C_{10}$ is $P_{FA}\cdot P[H_0]$.
• Write $C_{01}$ for cost of miss, i.e. cost when $H_1$ is true but decided $H_0$.
  • Probability of incurring cost $C_{01}$ is $P_{\text{Miss}}\cdot P[H_1]$.

Expected value of cost incurred

$$E[C]=P[A_1\mid H_0]\cdot P[H_0]\cdot C_{10}+P[A_0\mid H_1]\cdot P[H_1]\cdot C_{01}$$

MC design

Suppose we know:

• Both prior probabilities $P[H_0]$ and $P[H_1]$
• Both conditional distributions $P_{X|H_0}(x)$ and $P_{X|H_1}(x)$ (or $f_{X|H_0}(x)$ and $f_{X|H_1}(x)$)

The minimum cost (MC) design for a decision statistic $X$:

$$A_0=\text{set of x for which:}$$

Discrete case:

$$P_{X\mid H_0}(x)\cdot P[H_0]\cdot C_{10}\ge P_{X\mid H_1}(x)\cdot P[H_1]\cdot C_{01}$$

Continuous case:

$$f_{X\mid H_0}(x)\cdot P[H_0]\cdot C_{10}\ge f_{X\mid H_1}(x)\cdot P[H_1]\cdot C_{01}$$

Then $A_1=\{x\in \mathbb{R}\mid x\notin A_0\}$.

The MC design minimizes the expected value of the cost of error.

MC minimizes expected cost

Inside the argument that MAP minimizes total probability of error, we have this summation:

$$P_{\mathrm{ERR}}=\sum _{x\in A_1}{P}_{X|H_0}(x)\cdot P[H_0]+\sum _{x\in A_0}{P}_{X|H_1}(x)\cdot P[H_1]$$

The expected value of the cost has a similar summation:

$$E[C]=\sum _{x\in A_1}{P}_{X|H_0}(x)\cdot P[H_0]\cdot C_{10}+\sum _{x\in A_0}{P}_{X|H_1}(x)\cdot P[H_1]\cdot C_{01}$$

Following the same reasoning, we see that the cost is minimized if each $x$ is placed into $A_0$ precisely when the MC design condition is satisfied, and otherwise it is placed into $A_1$.

Link to original

Illustration

Example - MC Test: Smoke detector

MC Test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $\mathcal{N}(0,3^2)$.

Suppose that the background chance of smoke is $5\%$. Suppose the cost of a miss is $50\times$ the cost of a false alarm. Design an MC test for the alarm.

Compute the expected cost.

Solution

We have priors:

$$P[H_0]=0.95P[H_1]=0.05$$

And we have costs:

$$C_{10}=1C_{01}=50$$

(The ratio of these numbers is all that matters in the inequalities of the condition.)

The MC condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\cdot 0.95\cdot 1\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot 0.05\cdot 50\\ \\ \gg \gg e^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot \frac{2.5}{0.95}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2+\ln \left(\frac{2.5}{0.95}\right)\\ \\ \gg \gg x^2\stackrel{?}{\le }(x-8{)}^2-18\ln \left(\frac{2.5}{0.95}\right)\\ \\ \gg \gg x\le 2.91\end{array}$$

---

Therefore, $A_0$ is $x\le 2.91$, while $A_1$ is $x>2.91$.

The decision rule is: activate alarm when $x>2.91$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1\mid H_0]\\ \\ \gg \gg P[X>2.91\mid H_0]\gg \gg \approx 0.1660\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0\mid H_1]\\ \\ \gg \gg P[X\le 2.91]\gg \gg \approx 0.04488\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.95+P_{\text{Miss}}\cdot 0.05\approx 0.1599$$

---

PMF of total cost:

$$P_C(c)=\{\begin{array}{ll}0.002244& c=50\\ 0.1577& c=1\\ 0.840056& c=0\end{array}$$

Therefore $E[C]=0.27$.

Link to original

Mean square error

Theory - Minimum mean square error

Theory 1 - Minimum mean square error

Suppose our problem is to estimate or guess or predict the value of a random variable $X$ in a particular run of our experiment. Assume we have the distribution of $X$. Which value do we choose as our guess?

There is no single best answer to this question. The “best guess” number depends on additional factors in the problem context.

One method is to pick a value where the PMF or PDF of $X$ is maximal. This is a value of highest probability. There may be more than one!

Another method is to pick the expected value $E[X]$. This value may be impossible!

For the normal distribution, or any symmetrical distribution, these are the same value. For most distributions, though, they are not the same.

---

Mean square error (MSE)

Given some estimate $\hat{x}\in \mathbb{R}$ for a random variable $X$, the mean square error (MSE) of $\hat{x}$ is:

$$E[(X-\hat{x}{)}^2]$$

The MSE quantifies the typical (square of the) error. Error here means the difference between the true value $X$ and the estimate $\hat{x}$.

Other error estimates are reasonable and useful in niche contexts. For example, $E[|X-\hat{x}|]$ or $\mathrm{Max}|X-\hat{x}|$. They are not frequently used, so we do not consider their theory further.

---

In problem contexts where large errors are more costly than small errors (i.e. many real problems), the most likely value of $X$ (the point with maximal PDF) may fare poorly as an estimate.

It turns out that the expected value $E[X]$ also happens to be the value that minimizes the MSE.

Expected value minimizes MSE

Given a random variable $X$, its expected value $\hat{x}=E[X]$ is the estimate of $X$ with minimal mean square error.

The MSE error for $\hat{x}=E[X]$ is:

$$E[(X-\hat{x}{)}^2]=\mathrm{Var}[X]$$

Proof that $E[X]$ minimizes MSE

Expand the MSE error:

$$E[(X-\hat{x}{)}^2]\gg \gg E[X^2]-2\hat{x}E[X]+{\hat{x}}^2$$

Now minimize this parabola. Differentiate:

$$\frac{d}{d\hat{x}}E[(X-\hat{x}{)}^2]\gg \gg 0-2E[X]+2\hat{x}$$

Find zeros:

$$\begin{array}{c}0-2E[X]+2\hat{x}=0\\ \\ \gg \gg 2\hat{x}=2E[X]\gg \gg \hat{x}=E[x]\end{array}$$

---

When the estimate $\hat{x}$ is made in the absence of information (besides the distribution of $X$), it is called a blind estimate. Therefore, ${\hat{x}}_B=E[X]$ is the blind minimal MSE estimate, and $e_B=\mathrm{Var}[X]$ is the error of this estimate.

In the presence of additional information, for example that event $A$ is known, then the MSE estimate is ${\hat{x}}_A=E[X\mid A]$ and the error of this estimate is $e_{X|A}=\mathrm{Var}[X\mid A]$.

The MSE estimate can also be conditioned on another variable, say $Y$:

Minimal MSE of $X$ given $Y$

The minimal MSE estimate of $X$ given another variable $Y$:

$${\hat{x}}_M(y)=E[X\mid Y=y]$$

The error of this estimate is $\mathrm{Var}[X\mid Y=y]$, which equals $E[(X-{\hat{x}}_M(y){)}^2\mid Y=y]$.

Notice that the minimal MSE of $X$ given $Y$ can be used to define a random variable:

$${\hat{X}}_M(Y)=E[X\mid Y]={\hat{x}}_M(Y)$$

This is a derived variable from $Y$ given by composing $Y$ with the function ${\hat{x}}_M$.

The variable ${\hat{X}}_M(Y)$ provides the minimal MSE estimates of $X$ when experimental outcomes are viewed as providing the information of $Y$ only, and the model is used to derive estimates of $X$ from this information.

Link to original

Illustration

Example - Minimal MSE estimate given PMF, given fixed event

Minimal MSE estimate given PMF

Suppose $X$ has the following PMF:

$k$	1	2	3	4	5
$P_X(k)$	0.15	0.28	0.26	0.19	0.13

Find the minimal MSE estimate of $X$, given that $X$ is even. What is the error of this estimate?

Solution

The minimal MSE given $A$ is just $E[X\mid A]$ where $A=\{2,4\}$.

First compute the conditional PMF:

$$P_{X|A}(k)=\{\begin{array}{ll}0.19/0.47& k=4\\ 0.28/0.47& k=2\\ 0& k\ne 2,4\end{array}$$

Therefore:

$${\hat{x}}_A=2\frac{0.28}{0.47}+4\frac{0.19}{0.47}\approx 2.80851$$

The error is:

$$\begin{array}{c}{e}_{X|A}=(2-2.81{)}^2\frac{0.28}{0.47}+(4-2.81{)}^2\frac{0.19}{0.47}\\ \\ \gg \gg \approx 0.9633\end{array}$$ Link to original

Exercise - Minimal MSE estimate from joint PDF

Minimal MSE estimate from joint PDF

Here is the joint PDF of $X$ and $Y$:

$$f_{X,Y}=\{\begin{array}{ll}8xy& 0\le y\le x\le 1\\ 0& \text{otherwise}\end{array}$$

Find the minimal MSE estimate of $X$ in terms of $Y$.

What is the estimate of $X$ when $Y=0.2$? When $Y=0.8$?

Answer

${\hat{x}}_M(y)=\frac{2}{3}\cdot \frac{1-y^3}{1-y^2}$

${\hat{x}}_M(0.2)=0.6889{\hat{x}}_M(0.8)=0.9037$

Link to original

Theory - Line of minimal MSE

Theory 2 - Line of minimal MSE

Linear approximation is very common in applied math.

One could consider the linearization of ${\hat{x}}_M(y)$ (its tangent line at some point) instead of the exact function ${\hat{x}}_M(y)$.

One could instead minimize the MSE over all linear functions of $Y$. (Each of which is an RV.) The line with minimal MSE is called the linear estimator.

The difference here is:

• line of best fit at a single point vs.
• line of best fit over the whole range of $X$ and $Y$—weighted by likelihoods

Linear estimator: Line of minimal MSE

Let $L(y)$ be an arbitrary line $L(y)=ay+b$. Let us define ${\hat{X}}_L(Y)=L(Y)=aY+b$.

The mean square error (MSE) of this line is:

$$e_L(a,b)=E\left[\right(X-{\hat{X}}_L(Y){)}^2]$$

The linear estimator of $X$ in terms of $Y$ is the line $L^{\star }$ with minimal MSE, and it is:

$$\begin{array}{c}{L}^{\star }(y)=a^{\star }y+b^{\star }\\ \\ a^{\star }={\rho }_{X,Y}\frac{{\sigma }_X}{{\sigma }_Y},b^{\star }={\mu }_X-a^{\star }{\mu }_Y\end{array}$$

The error value at the (best) linear estimator, $e_{L_{\min }}$, is:

$$e_{L_{\min }}={\sigma }_X^2(1-{\rho }_{X,Y}^2)$$

Theorem: The error variable of the linear estimator, $X-{\hat{X}}_{L_{\min }}(Y)$, is perfectly uncorrelated with $Y$.

Slope and ${\rho }_{X,Y}$

Notice:

$$\frac{{\hat{X}}_{L_{\min }}(Y)-{\mu }_X}{{\sigma }_X}={\rho }_{X,Y}\cdot \left(\frac{Y-{\mu }_Y}{{\sigma }_Y}\right)$$

Thus, for standardized variables $X$ and $Y$, it turns out ${\rho }_{X,Y}$ is the slope of the linear estimator.

In each graph, $E[X]=E[Y]=0$ and $\mathrm{Var}[X]=\mathrm{Var}[Y]=1$.

The line of minimal MSE is the “best fit” line, ${\hat{X}}_{L_{\min }}(Y)={\rho }_{X,Y}Y$.

Link to original

Illustration

Example - Estimating on a variable interval

Estimating on a variable interval

Suppose that $R\sim \mathrm{Unif}(0,1)$ and suppose $X\sim \mathrm{Unif}(0,R)$.

(a) Find ${\hat{x}}_M(r)$ (b) Find ${\hat{r}}_M(x)$ (c) Find ${\hat{R}}_{L_{\min }}(X)$

Solution

(a) Find ${\hat{x}}_M(r)$:

We know ${\hat{x}}_M(r)=E[X\mid R=r]$.

Given $R=r$, so $X$ is uniform on $(0,r)$, we have $E[X\mid R=r]=\frac{r}{2}$.

$${\hat{x}}_M(r)=\frac{r}{2}$$

---

(b) Find ${\hat{r}}_M(x)$:

We know ${\hat{r}}_M(x)=E[R\mid X=x]$.

We know $f_R$ and $f_{X|R}$. From these we derive the joint distribution $f_{X,R}$:

$$f_R(r)=\{\begin{array}{ll}1& r\in (0,1)\\ 0& \text{otherwise}\end{array}{f}_{X|R}(x|r)=\{\begin{array}{ll}1/r& x\in (0,r)\\ 0& \text{otherwise}\end{array}$$ $$\gg \gg f_{X,R}(x,r)=f_{X|R}\cdot f_R=\{\begin{array}{ll}1/r& 0<x<r<1\\ 0& \text{else}\end{array}$$

Now extract the marginal $f_X$:

$$\begin{array}{c}\gg \gg f_X(x)={\int }_{-\infty }^{\infty }{f}_{X,R}(x,r)dr\\ \\ \gg \gg {\int }_x^1\frac{1}{r}dr\gg \gg -\ln x(0<x<1)\end{array}$$

Now deduce the conditional $f_{R|X}$:

$$f_{R|X}=\frac{f_{X,R}}{f_X}=\{\begin{array}{ll}\frac{-1}{r\ln x}& 0<x<r<1\\ 0& \text{otherwise}\end{array}$$

Then:

$$\begin{array}{c}E[R\mid X=x]\gg \gg {\int }_x^{1}r\frac{-1}{r\ln x}dr\\ \\ \gg \gg {\hat{r}}_M(x)=\frac{x-1}{\ln x}\end{array}$$

---

(c) Find ${\hat{R}}_{L_{\min }}(X)$:

We need all the basic statistics.

$E[R]=1/2$ because $R\sim \mathrm{Unif}((0,1))$.

${\sigma }_R^2=\frac{(b-a{)}^2}{12}=1/12$.

$E[X]=1/4$ using the marginal PDF $f_X(x)=-\ln x$ on $x\in (0,1)$. (IBP and L’Hopital are needed.)

${\sigma }_X=\sqrt{7}/12$ also using the marginal $f_X(x)=-\ln x$.

$E[XR]=1/6$ using $f_{X,R}(x,r)$, namely:

$$\begin{array}{c}E[XR]={\int }_{r=0}^1{\int }_{x=0}^{r}xr\frac{1}{r}dxdr\\ \\ \gg \gg {\int }_0^1\frac{x^2}{2}dx\gg \gg \frac{1}{6}\end{array}$$

From all this we infer $\mathrm{Cov}[X,R]=1/24$ and ${\rho }_{X,R}=\sqrt{3/7}$.

Hence:

$$L_{\min }(x)=\frac{6}{7}x+\frac{2}{7}$$

Thus:

$${\hat{R}}_{L_{\min }}(X)=\frac{6}{7}X+\frac{2}{7}$$

Link to original

Exercise - Line of minimal MSE given joint PDF

Line of minimal MSE given joint PDF

Here is the joint PDF of $X$ and $Y$:

$$f_{X,Y}=\{\begin{array}{ll}8xy& 0\le y\le x\le 1\\ 0& \text{otherwise}\end{array}$$

Find the line giving the linear MSE estimate of $X$ in terms of $Y$.

What is the expected error of this line, $e_{L_{\min }}$?

What is the estimate of $X$ when $Y=0.2$? When $Y=0.8$?

Answer

${\hat{X}}_{L_{\min }}(Y)=0.3637Y+0.6060$

$e_{L_{\min }}=0.02020$

${\hat{x}}_{L_{\min }}(0.2)=0.67874{\hat{x}}_{L_{\min }}(0.8)=0.89696$

Link to original