Problems

01

Linear least-squares: non-orthogonal columns

Solve the linear least-squares problem for the following inconsistent system:

$$\left(\begin{array}{rr}1& -2\\ -1& 2\\ 0& 3\\ 2& 5\end{array}\right)\mathbf{x}=\left(\begin{array}{r}3\\ 1\\ -4\\ 2\end{array}\right).$$

Compute the error to two decimal places.

02

Linear least-squares: orthogonal columns

Solve the linear least-squares problem for the following inconsistent system, using the fact that the columns of $A$ are orthogonal:

$$\left(\begin{array}{rrr}4& 0& 1\\ 1& -5& 1\\ 6& 1& 0\\ 1& -1& -5\end{array}\right)\mathbf{x}=\left(\begin{array}{r}9\\ 0\\ 0\\ 0\end{array}\right).$$

Compute the error to two decimal places.

03

Fitting a plane to four points

Consider the four data points in $3$D space:

$$(x,y,z)=(1,0,0),(0,1,1),(-1,0,3),(0,-1,4).$$

Find the parameters $a,b,c$ for which the plane defined by $z=a+bx+cy$ best fits these data points.

You may notice that the data points correspond to heights $z=0,1,3,4$ at the four corners of a square. Normally a plane is defined by 3 points (non-colinear) through which it passes. No plane passes through these four given points. But there is a plane that minimizes the sum of squares of vertical errors.

04

LLS summation curve model

Consider the data points: $(x,y)=(1,7.9),(2,5.4),(3,-0.9)$.
Find the parameters of best fit for the model $y=A\cos x+B\sin x$. Clearly identify your design matrix, observation vector, and parameter vector. Compute the error vector and the total error.

05

Penrose quasi-inverse

For this problem, use the definition of $A^{+}$ based on the SVD of $A$.

• (a) Verify that $A{A}^{+}A=A$ and that $A^{+}A{A}^{+}=A^{+}$ for the example $A=\left(\begin{array}{cc}-3& 1\\ 6& -2\\ 6& -2\end{array}\right)$ studied in the section on quasi-inverse. Think about how and why this happens in terms of the SVD.
• (b) Show that for any $A$ matrix, $A{A}^{+}\mathbf{v}$ is the projection of $\mathbf{v}$ onto the image of $A$.
• (c) Show that for any $A$ matrix, $A{A}^{+}A=A$ and $A^{+}A{A}^{+}=A^{+}$.

It may be helpful (though not required) for this problem to use the presentation of SVD in the form $A={\sigma }_1{\mathbf{u}}_1{\mathbf{v}}_1^{\mathsf{T}}+{\sigma }_2{\mathbf{u}}_2{\mathbf{v}}_2^{\mathsf{T}}+\cdots$ as in the definition of the SVD. In order to use this, figure out what the corresponding presentation of the SVD of $A^{+}$ should be.

06

LLS uniqueness

• (a) Show that: $A\mathbf{x}=0$ if and only if $A^{\mathsf{T}}A\mathbf{x}=0$. (Hint: use the fact that ${\mathbf{x}}^{\mathsf{T}}{A}^{\mathsf{T}}A\mathbf{x}=|A\mathbf{x}{|}^2$.)
• (b) Show that: $A^{\mathsf{T}}A$ is invertible if and only if $A$ has independent columns. (Warning: $A$ need not be square. Hint: consider the dimensions of kernels, and use (a) as well as the rank-nullity theorem.)
• (c) Explain why the LLS problem for $A$ and $\mathbf{b}$ has a unique solution if and only if the columns of $A$ are independent. (Use (b).)
• (d) Continuing from (b), show that: $\mathrm{rank}(A)=\mathrm{rank}(A^{\mathsf{T}}A)$. (Hint: apply rank-nullity to both matrices. Notice that $A$ and $A^{\mathsf{T}}A$ have the same number of columns.)

You may observe that (a) implies $\dim \mathrm{Ker}(A)=\dim \mathrm{Ker}(A^{\mathsf{T}}A)$, while (d) means $\dim \mathrm{Im}(A)=\dim \mathrm{Im}(A^{\mathsf{T}}A)$. In other words, $A$ and $A^{\mathsf{T}}A$ always have the same size kernels and the same size images. This makes sense in terms of the SVD if you think about kernels, cokernels, and images generated by the orthonormal basis vectors ${\mathbf{u}}_i$’s and ${\mathbf{v}}_i$’s, writing $A={\sigma }_1{\mathbf{u}}_1{\mathbf{v}}_1^{\mathsf{T}}+{\sigma }_2{\mathbf{u}}_2{\mathbf{v}}_2^{\mathsf{T}}+\cdots$ as in the previous problem (keeping careful track of which ${\sigma }_i$’s are zero or nonzero!).