Examples

Finding eigenvectors and eigenvalues

Finding eigenvectors and eigenvalues

Problem: Find the eigenvectors and their eigenvalues for the matrix $A=\left(\begin{array}{rrr}4& 0& 1\\ -2& 1& 0\\ -2& 0& 1\end{array}\right)$.
Solution: First find the eigenvalues:

$$\begin{array}{rl}\det A_{\lambda }& =\left|\begin{array}{ccc}4-\lambda & 0& 1\\ -2& 1-\lambda & 0\\ -2& 0& 1-\lambda \end{array}\right|\\ & \\ & =(4-\lambda )\left|\begin{array}{cc}1-\lambda & 0\\ 0& 1-\lambda \end{array}\right|-0\left|\begin{array}{cc}-2& 0\\ -2& 1-\lambda \end{array}\right|+1\left|\begin{array}{cc}-2& 1-\lambda \\ -2& 0\end{array}\right|\\ & \\ & =(4-\lambda )(1-\lambda {)}^2-0+2(1-\lambda )\\ & =-{\lambda }^3+6{\lambda }^2-11\lambda +6.\end{array}$$

The roots are $\lambda =1,2,3$. (The rational roots theorem leads to the guess $\lambda =2$, and then you can divide the cubic polynomial by $\lambda -2$ to obtain a quadratic one, which is easily factored.)

Now we seek eigenvectors by solving $A_{\lambda }\mathbf{x}=0$ for each $\lambda$ root. When $\lambda =1$:

$$A_{\lambda }\mathbf{x}=\left(\begin{array}{rrr}3& 0& 1\\ -2& 0& 0\\ -2& 0& 0\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ 0\end{array}\right).$$

To solve for $\mathbf{x}$, row reduce $A_{\lambda }$ to obtain a matrix in RREF:

$$\begin{array}{rl}{A}_{\lambda }& =\left(\begin{array}{rrr}3& 0& 1\\ -2& 0& 0\\ -2& 0& 0\end{array}\right)\sim \left(\begin{array}{rrr}3& 0& 1\\ 0& 0& 2/3\\ 0& 0& 2/3\end{array}\right)\\ & \\ & \sim \left(\begin{array}{rrr}3& 0& 0\\ 0& 0& 2/3\\ 0& 0& 0\end{array}\right)\sim \left(\begin{array}{rrr}1& 0& 0\\ 0& 0& 1\\ 0& 0& 0\end{array}\right).\end{array}$$

So we know $x_1=x_3=0$ and $x_2$ is free. Plugging these in we have $\mathbf{x}=\left(\begin{array}{c}0\\ x_2\\ 0\end{array}\right)$, and we choose $x_2=1$ obtaining $\left(\begin{array}{c}0\\ 1\\ 0\end{array}\right)$ for an eigenvector.

Next for $\lambda =2$ we have:

$$A_{\lambda }\mathbf{x}=\left(\begin{array}{rrr}2& 0& 1\\ -2& -1& 0\\ -2& 0& -1\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ 0\end{array}\right).$$

To solve for $\mathbf{x}$, row reduce $A_{\lambda }$ to obtain a matrix in RREF:

$$\begin{array}{rl}{A}_{\lambda }& =\left(\begin{array}{rrr}2& 0& 1\\ -2& -1& 0\\ -2& 0& -1\end{array}\right)\sim \left(\begin{array}{rrr}2& 0& 1\\ 0& -1& 1\\ 0& 0& 0\end{array}\right)\sim \left(\begin{array}{rrr}1& 0& 1/2\\ 0& 1& -1\\ 0& 0& 0\end{array}\right).\end{array}$$

So we know $x_1=-\frac{1}{2}{x}_3$ and $x_2=+x_3$. Plugging these in we have $\mathbf{x}=\left(\begin{array}{c}-\frac{1}{2}{x}_3\\ x_3\\ x_3\end{array}\right)$, and we choose $x_3=2$ obtaining $\left(\begin{array}{c}-1\\ 2\\ 2\end{array}\right)$.

Finally for $\lambda =3$ we have:

$$A_{\lambda }\mathbf{x}=\left(\begin{array}{rrr}1& 0& 1\\ -2& -2& 0\\ -2& 0& -2\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ 0\end{array}\right).$$

To solve for $\mathbf{x}$, row reduce $A_{\lambda }$ to obtain a matrix in RREF:

$$\begin{array}{rl}{A}_{\lambda }& =\left(\begin{array}{rrr}1& 0& 1\\ -2& -2& 0\\ -2& 0& -2\end{array}\right)\sim \left(\begin{array}{rrr}1& 0& 1\\ 0& -2& 2\\ 0& 0& 0\end{array}\right)\sim \left(\begin{array}{rrr}1& 0& 1\\ 0& 1& -1\\ 0& 0& 0\end{array}\right).\end{array}$$

So we know $x_1=-x_3$ and $x_2=+x_3$. Plugging these in we have $\mathbf{x}=\left(\begin{array}{c}-x_3\\ x_3\\ x_3\end{array}\right)$, and we choose $x_3=1$ obtaining $\left(\begin{array}{c}-1\\ 1\\ 1\end{array}\right)$.

Changing to eigenbasis yields diagonal matrix

Changing to eigenbasis yields diagonal matrix

Notice that $A$ in the previous example has three eigenvectors, and that they are independent. Therefore they constitute a basis of ${\mathbb{R}}^3$; let’s call this basis $\mathcal{C}$. Putting these basis vectors as the columns of a matrix, we create the change of basis transfer matrix from $\mathcal{C}$ to the standard basis $\mathcal{B}$:

$${}_{\mathcal{B}}{T}_{\mathcal{C}}=\left(\begin{array}{ccc}{\mathbf{c}}_1& {\mathbf{c}}_2& {\mathbf{c}}_3\end{array}\right)=\left(\begin{array}{rcc}0& -1& -1\\ 1& 2& 1\\ 0& 2& 1\end{array}\right).$$

So ${}_{\mathcal{B}}{T}_{\mathcal{C}}\cdot [\mathbf{x}{]}_{\mathcal{C}}=x_1{\mathbf{c}}_1+x_2{\mathbf{c}}_2+x_3{\mathbf{c}}_3=x_1\left(\begin{array}{c}0\\ 1\\ 0\end{array}\right)+x_2\left(\begin{array}{c}-1\\ 2\\ 2\end{array}\right)+x_3\left(\begin{array}{c}-1\\ 1\\ 1\end{array}\right)$.

Now consider that these are eigenvectors of $A$ with eigenvalues $1,2,3$:

$$A{\mathbf{c}}_1={\mathbf{c}}_1,A{\mathbf{c}}_2=2{\mathbf{c}}_2,A{\mathbf{c}}_3=3{\mathbf{c}}_3.$$

If we write $A$ in the basis $\mathcal{C}$ using the transfer matrix ${}_{\mathcal{B}}{T}_{\mathcal{C}}$, we have:

$$\begin{array}{rl}[A{]}_{\mathcal{C}}& ={}_{\mathcal{C}}{T}_{\mathcal{B}}\cdot A\cdot {}_{\mathcal{B}}{T}_{\mathcal{C}}\\ & \\ & ={\left(\begin{array}{rcc}0& -1& -1\\ 1& 2& 1\\ 0& 2& 1\end{array}\right)}^{-1}\left(\begin{array}{rrr}4& 0& 1\\ -2& 1& 0\\ -2& 0& 1\end{array}\right)\left(\begin{array}{rcc}0& -1& -1\\ 1& 2& 1\\ 0& 2& 1\end{array}\right)=\left(\begin{array}{rcc}1& 0& 0\\ 0& 2& 0\\ 0& 0& 3\end{array}\right).\end{array}$$

In other words, the matrix of $A$ is a diagonal matrix when written in the basis of its own eigenvectors.

Matrix with too few eigenvectors

Matrix with too few eigenvectors

The matrix $A=\left(\begin{array}{rc}1& 1\\ 0& 1\end{array}\right)$ has $\det A_{\lambda }=(1-\lambda {)}^2$. Therefore it has a “double” eigenvalue $\lambda =+1$. To find the eigenvectors:

$$A_{+1}\cdot \mathbf{x}=\left(\begin{array}{cc}0& 1\\ 0& 0\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right).$$

The matrix is already in RREF. So the equation says that $x_1$ is free and $x_2=0$. Therefore we have the eigenvector $\left(\begin{array}{c}1\\ 0\end{array}\right)$.

However, no more eigenvectors are available (other than scalar multiples of this one, which would not be independent of this one). Therefore it is impossible to find a basis of eigenvectors, and it is thus impossible to convert $A$ to diagonal form by a change of basis.

Matrix with complex eigenvalues and eigenvectors

Matrix with complex eigenvalues and eigenvectors

The matrix $A=\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)$ has $\det A_{\lambda }={\lambda }^2+1$ so its eigenvalues are $\lambda =\pm i$. To find the eigenvectors:

$$A_i\cdot \mathbf{x}=\left(\begin{array}{cc}-i& -1\\ 1& -i\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right).$$

Row reduce the matrix $A_i$:

$$A_i=\left(\begin{array}{cc}-i& -1\\ 1& -i\end{array}\right)\sim \left(\begin{array}{cc}-i& -1\\ 0& 0\end{array}\right)\sim \left(\begin{array}{cc}1& -i\\ 0& 0\end{array}\right).$$

This is now in RREF, so we see that $x_1=i{x}_2$ and $x_2$ is free, so we get the eigenvector $\left(\begin{array}{c}i\\ 1\end{array}\right)$.

Following the same procedure for $\lambda =-i$, we find the eigenvector $\left(\begin{array}{c}-i\\ 1\end{array}\right)$.

Computing SVD by hand

Computing SVD by hand

Problem: Find an SVD for the matrix $A=\left(\begin{array}{rr}1& -1\\ -2& 2\\ 2& -2\end{array}\right)$.

Solution: First calculate that $A^{\mathsf{T}}A=\left(\begin{array}{rc}9& -9\\ -9& 9\end{array}\right)$ and this matrix has $\lambda =18$ and $\lambda =0$ eigenvalues. Therefore we set ${\sigma }_1=3\sqrt{2}$ and ${\sigma }_2=0$. By solving $\left(\begin{array}{rr}-9& -9\\ -9& -9\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$ we obtain an eigenvector $\left(\begin{array}{r}1\\ -1\end{array}\right)$ which we renormalize to the unit vector $\left(\begin{array}{r}1/\sqrt{2}\\ -1/\sqrt{2}\end{array}\right)$. By solving $\left(\begin{array}{rr}9& -9\\ -9& 9\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$ we obtain an eigenvector $\left(\begin{array}{r}1\\ 1\end{array}\right)$ which we renormalize to the unit vector $\left(\begin{array}{r}1/\sqrt{2}\\ 1/\sqrt{2}\end{array}\right)$. We now have $V$ and $\Sigma$:

$$\Sigma =\left(\begin{array}{cc}3\sqrt{2}& 0\\ 0& 0\\ 0& 0\end{array}\right),V=\left(\begin{array}{rr}1/\sqrt{2}& 1/\sqrt{2}\\ -1/\sqrt{2}& 1/\sqrt{2}\end{array}\right).$$

Set

$${\mathbf{u}}_1={\sigma }_1^{-1}A{\mathbf{v}}_1=\frac{1}{3\sqrt{2}}\left(\begin{array}{r}2/\sqrt{2}\\ -4/\sqrt{2}\\ 4/\sqrt{2}\end{array}\right)=\left(\begin{array}{r}1/3\\ -2/3\\ 2/3\end{array}\right)$$

and we notice that $|{\mathbf{u}}_1|=1$. Now we need ${\mathbf{u}}_2,{\mathbf{u}}_3$ unit vectors orthogonal to each other and to ${\mathbf{u}}_1$. We first calculate a span of the kernel of the matrix ${\mathbf{u}}_1^{\mathsf{T}}$, and then perform Gram-Schmidt to orthogonalize it. For the kernel:

$$\left(\begin{array}{ccc}1/3& -2/3& 2/3\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)=0,$$

and therefore $x_1-2x_2+2x_3=0$ so we can set $x_2,x_3$ free and solve for $x_1$, thereby obtaining generators for the kernel:

$$\ker {\mathbf{u}}_1^{\mathsf{T}}=\{\left(\begin{array}{c}2x_2-2x_3\\ x_2\\ x_3\end{array}\right)\}=⟨{\mathbf{w}}_2=\left(\begin{array}{c}2\\ 1\\ 0\end{array}\right),{\mathbf{w}}_3=\left(\begin{array}{c}-2\\ 0\\ 1\end{array}\right)⟩.$$

Now we already have ${\mathbf{w}}_2\perp {\mathbf{u}}_1$ by our method of defining ${\mathbf{w}}_2$. So it just remains to do the last step of Gram-Schmidt. We calculate that

$${\mathbf{w}}_3-\frac{{\mathbf{w}}_3\cdot {\mathbf{u}}_1}{{\mathbf{u}}_1\cdot {\mathbf{u}}_1}{\mathbf{u}}_1-\frac{{\mathbf{w}}_3\cdot {\mathbf{w}}_2}{{\mathbf{w}}_2\cdot {\mathbf{w}}_2}{\mathbf{w}}_2=\left(\begin{array}{c}-2\\ 0\\ 1\end{array}\right)-\frac{0}{1}\left(\begin{array}{r}1/3\\ -2/3\\ 2/3\end{array}\right)-\frac{-4}{5}\left(\begin{array}{c}2\\ 1\\ 0\end{array}\right)=\left(\begin{array}{c}-2/5\\ 4/5\\ 1\end{array}\right).$$

Then

$${\mathbf{u}}_2=\frac{1}{|{\mathbf{w}}_2|}{\mathbf{w}}_2=\left(\begin{array}{c}2/\sqrt{5}\\ 1/\sqrt{5}\\ 0\end{array}\right),{\mathbf{u}}_3=\frac{5}{\sqrt{45}}\left(\begin{array}{c}-2/5\\ 4/5\\ 1\end{array}\right)=\left(\begin{array}{r}-2/\sqrt{45}\\ 4/\sqrt{45}\\ 5/\sqrt{45}\end{array}\right).$$

We have finished, and the SVD is given by:

$$A=U\Sigma V^{\mathsf{T}}=\left(\begin{array}{rcr}1/3& 2/\sqrt{5}& -2/\sqrt{45}\\ -2/3& 1/\sqrt{5}& 4/\sqrt{45}\\ 2/3& 0& 5/\sqrt{45}\end{array}\right)\left(\begin{array}{cc}3\sqrt{2}& 0\\ 0& 0\\ 0& 0\end{array}\right)\left(\begin{array}{rr}1/\sqrt{2}& -1/\sqrt{2}\\ 1/\sqrt{2}& 1/\sqrt{2}\end{array}\right).$$