W13 - Examples

Significance testing

One-tail test: Weighted die

One-tail test: Weighted die

Your friend gives you a single regular die, and say she is worried that it has been weighted to prefer the outcome of 2. She wants you to test it.

Design a significance test for the data of 20 rolls of the die to determine whether the die is weighted. Use significance level $\alpha =0.05$.

Solution

Let $X$ count the number of 2s that come up.

The Claim: “the die is weighted to prefer 2”
The null hypothesis $H_0$: “the die is normal”

Assuming $H_0$ is true, then $X\sim \mathrm{Bin}(20,1/6)$, and therefore:

$$P_{X|H_0}(k)=\left(\genfrac{}{}{0ex}{}{20}{k}\right)(1/6{)}^k(5/6{)}^{20-k}$$

---

⚠️ Notice that “prefer 2” implies the claim is for more 2s than normal.

Therefore: Choose a one-tail rejection region.

Need $r$ such that:

$$\begin{array}{rl}P[X\ge r\mid H_0]& =0.05\\ & \\ ⟺P[X<r\mid H_0]& =0.95\end{array}$$

---

Solve for $r$ by computing conditional CDF values:

$k:$	0	1	2	3	4	5	6	7
$F_{X\mid H_0}(k):$	0.026	0.130	0.329	0.567	0.769	0.898	0.963	0.989

Therefore, choose $r=6$:

$P[X\ge 6\mid H_0]<0.04$, but $P[X\ge 5\mid H_0]>0.05$. Final answer:

$$R=\{x\mid x\ge 6\}$$ Link to original

Two-tail test: Circuit voltage

Two-tail test: Circuit voltage

A boosted AC circuit is supposed to maintain an average voltage of $130\mathrm{V}$ with a standard deviation of $2.1\mathrm{V}$. Nothing else is known about the voltage distribution.

Design a two-tail test incorporating the data of 40 independent measurements to determine if the expected value of the voltage is truly $130\mathrm{V}$. Use $\alpha =0.02$.

Solution

Use $M_{40}(V)$ as the decision statistic, i.e. the sample mean of 40 measurements of $V$.

The Claim to test: $E[V]\ne 130$

The null hypothesis $H_0$: $E[V]=130$

Rejection region:

$$|M_{40}-130|\ge c$$

where $c$ is chosen so that $P[|M_{40}-130|\ge c]=0.02$

---

Assuming $H_0$, we expect that:

$$E[M_{40}]=130,{\sigma }_{M_{40}}^2=\frac{2.1^2}{40}\approx 0.110$$

Recall Chebyshev’s inequality:

$$P[|M_{40}-130|\ge c]\le \frac{{\sigma }_{M_{40}}^2}{c^2}\approx \frac{0.110}{c^2}$$

---

Now solve:

$$\frac{0.110}{c^2}=0.02\gg \gg c\approx 2.348$$

Therefore the rejection region should be:

$$M_{40}<127.65\cup 132.35<M_{40}$$ Link to original

One-tail test with a Gaussian: Weight loss drug

One-tail test with a Gaussian: Weight loss drug

Assume that in the background population in a specific demographic, the distribution of a person’s weight $W$ satisfies $W\sim \mathcal{N}(190,24^2)$. Suppose that a pharmaceutical company has developed a weight-loss drug and plans to test it on a group of 64 individuals.

Design a test at the $\alpha =0.01$ significance level to determine whether the drug is effective.

Solution

Since the drug is tested on 64 individuals, we use the sample mean $M_{64}(W)$ as the decision statistic.

The Claim: “the drug is effective in reducing weight”

The null hypothesis $H_0$: “no effect: weights on the drug still follow $\mathcal{N}(190,24^2)$”

Assuming $H_0$ is true, then $W\sim \mathcal{N}(190,24^2)$.

⚠️ One-tail test because the drug is expected to reduce weight (unidirectional). Rejection region:

$$M_{64}(W)\le r$$

---

Calculate:

$${\sigma }_{M_{64}}^2\gg \gg \frac{24^2}{64}\gg \gg 9$$

⚠️ Standardized $M_{64}(W)$ is approximately normal!

(The standardization of $M_{64}(W)$ removes the effect of $\frac{1}{n}$. As if it’s the summation.)

So, standardize and apply CLT:

$$\begin{array}{c}\frac{M_{64}(W)-190}{\sqrt{9}}\sim \mathcal{N}(0,1),\\ \\ \gg \gg P[M_{64}(W)\le r]\approx P\left[Z\le \frac{r-190}{3}\right]=\Phi \left(\frac{r-190}{3}\right)\end{array}$$

---

Solve:

$$\begin{array}{c}P[M_{64}(W)\le r]=0.01\\ \\ \gg \gg \Phi \left(\frac{r-190}{3}\right)=0.01\\ \\ \gg \gg \Phi \left(\frac{190-r}{3}\right)=0.99\\ \\ \gg \gg \frac{190-r}{3}=2.33\\ \\ \gg \gg r=183.01\end{array}$$

Therefore, the rejection region:

$$M_{64}(W)\le 183.01$$Link to original

Binary hypothesis testing

ML test: Smoke detector

ML test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $\mathcal{N}(0,3^2)$.

Design an ML test for the detector electronics to decide whether to activate the alarm.

What are the three error probabilities? (Type I, Type II, Total.)

Solution

First, establish the conditional distributions:

$$X\mid H_0\sim \mathcal{N}(0,3^2)X\mid H_1\sim \mathcal{N}(8,3^2)$$

Density functions:

$$f_{X|H_0}=\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}{f}_{X|H_1}=\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}$$

---

The ML condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2\\ \\ \gg \gg x^2\stackrel{?}{\le }(x-8{)}^2\\ \\ \gg \gg x\le 4\end{array}$$

---

Therefore, $A_0$ is $x\le 4$, while $A_1$ is $x>4$.

The decision rule is: activate alarm when $x>4$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1\mid H_0]\gg \gg P[X>4\mid H_0]\\ \\ \gg \gg 1-P\left[\frac{X-0}{3}\le \frac{4}{3}|H_0\right]\\ \\ \gg \gg 1-P[Z\le 1.3333]\gg \gg \approx 0.0912\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0\mid H_1]\gg \gg P[X\le 4\mid H_1]\\ \\ \gg \gg P\left[\frac{X-8}{3}\le \frac{4-8}{3}|H_1\right]\\ \\ \gg \gg P[Z\le -1.3333]\gg \gg \approx 0.0912\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.5+P_{\text{Miss}}\cdot 0.5\gg \gg \approx 0.0912$$ Link to original

MAP test: Smoke detector

MAP test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $\mathcal{N}(0,3^2)$.

Suppose that the background chance of smoke is $5\%$. Design a MAP test for the alarm.

What are the three error probabilities? (Type I, Type II, Total.)

Solution

First, establish priors:

$$P[H_0]=0.95P[H_1]=0.05$$

The MAP condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\cdot 0.95\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot 0.05\\ \\ \gg \gg e^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot \frac{0.05}{0.95}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2+\ln \left(\frac{0.05}{0.95}\right)\\ \\ \gg \gg x^2\stackrel{?}{\le }(x-8{)}^2-18\ln \left(\frac{0.05}{0.95}\right)\\ \\ \gg \gg x\le 7.31\end{array}$$

---

Therefore, $A_0$ is $x\le 7.31$, while $A_1$ is $x>7.31$.

The decision rule is: activate alarm when $x>7.31$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1\mid H_0]\gg \gg P[X>7.31\mid H_0]\\ \\ \gg \gg 1-P[Z\le 2.4367]\gg \gg \approx 0.007411\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0\mid H_1]\gg \gg P[X\le 7.31\mid H_1]\\ \\ \gg \gg P[Z\le -0.23]\gg \gg \approx 0.4090\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.95+P_{\text{Miss}}\cdot 0.05\approx 0.02749$$ Link to original

MC Test: Smoke detector

MC Test: Smoke detector

Suppose that a smoke detector sensor is configured to produce $8\mathrm{V}$ when there is smoke, and $0\mathrm{V}$ otherwise. But there is background noise with distribution $\mathcal{N}(0,3^2)$.

Suppose that the background chance of smoke is $5\%$. Suppose the cost of a miss is $50\times$ the cost of a false alarm. Design an MC test for the alarm.

Compute the expected cost.

Solution

We have priors:

$$P[H_0]=0.95P[H_1]=0.05$$

And we have costs:

$$C_{10}=1C_{01}=50$$

(The ratio of these numbers is all that matters in the inequalities of the condition.)

The MC condition becomes:

$$\begin{array}{c}\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\cdot 0.95\cdot 1\stackrel{?}{\ge }\frac{1}{\sqrt{2\pi 9}}{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot 0.05\cdot 50\\ \\ \gg \gg e^{-\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2}\stackrel{?}{\ge }{e}^{-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2}\cdot \frac{2.5}{0.95}\\ \\ \gg \gg -\frac{1}{2}{\left(\frac{x-0}{3}\right)}^2\stackrel{?}{\ge }-\frac{1}{2}{\left(\frac{x-8}{3}\right)}^2+\ln \left(\frac{2.5}{0.95}\right)\\ \\ \gg \gg x^2\stackrel{?}{\le }(x-8{)}^2-18\ln \left(\frac{2.5}{0.95}\right)\\ \\ \gg \gg x\le 2.91\end{array}$$

---

Therefore, $A_0$ is $x\le 2.91$, while $A_1$ is $x>2.91$.

The decision rule is: activate alarm when $x>2.91$.

---

Type I error:

$$\begin{array}{c}{P}_{FA}=P[A_1\mid H_0]\\ \\ \gg \gg P[X>2.91\mid H_0]\gg \gg \approx 0.1660\end{array}$$

Type II error:

$$\begin{array}{c}{P}_{\text{Miss}}=P[A_0\mid H_1]\\ \\ \gg \gg P[X\le 2.91]\gg \gg \approx 0.04488\end{array}$$

Total error:

$$P_{\mathrm{ERR}}=P_{FA}\cdot 0.95+P_{\text{Miss}}\cdot 0.05\approx 0.1599$$

---

PMF of total cost:

$$P_C(c)=\{\begin{array}{ll}0.002244& c=50\\ 0.1577& c=1\\ 0.840056& c=0\end{array}$$

Therefore $E[C]=0.27$.

Link to original