Theory

Introduction

Vectors can be considered from an abstract perspective as items in a space of vectors all of which satisfy the rules of the “game” of vectors, as well as from a concrete perspective as lists of numbers upon which the operations of vectors are defined (addition and scaling).

Similarly, matrices can be considered from an abstract perspective as “linear operations” upon an abstract space of vectors, or from a concrete perspective as arrays of numbers which operate upon the lists of numbers in concrete vectors.

Matrices acting on vectors

An $n\times m$ matrix is an $n\times m$ array of numbers. Given a matrix called $A$, the numbers making it up are called “coefficients” and written $a_{ij}$ for $i=1,\dots ,n$ and $j=1,\dots ,m$. Sometimes we write $A=(a_{ij})$. The first index always specifies the row, and the second index specifies the column. For example, we could have:

$$A=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\text{or}B=\left(\begin{array}{ccc}{b}_{11}& b_{12}& b_{13}\\ b_{21}& b_{22}& b_{23}\\ b_{31}& b_{32}& b_{33}\end{array}\right).$$

Suppose we have a 2D vector $\mathbf{u}$ and a 3D vector $\mathbf{v}$:

$$\mathbf{u}=\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right)\text{and}\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\end{array}\right).$$

Now the action of $A$ on $\mathbf{u}$ and of $B$ on $\mathbf{v}$ can be written out with these formulas:

$$A\mathbf{u}=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right)=\left(\begin{array}{c}{a}_{11}{u}_1+a_{12}{u}_2\\ a_{21}{u}_1+a_{22}{u}_2\end{array}\right),$$ $$B\mathbf{v}=\left(\begin{array}{ccc}{b}_{11}& b_{12}& b_{13}\\ b_{21}& b_{22}& b_{23}\\ b_{31}& b_{32}& b_{33}\end{array}\right)\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\end{array}\right)=\left(\begin{array}{c}{b}_{11}{v}_1+b_{12}{v}_2+b_{13}{v}_3\\ b_{21}{v}_1+b_{22}{v}_2+b_{23}{v}_3\\ b_{31}{v}_1+b_{32}{v}_2+b_{33}{v}_3\end{array}\right).$$

Matrices acting on vectors (part 2)

Let $C$ be an $n\times m$ matrix and let $\mathbf{w}$ be an $m$D vector. Using summation notation for large numbers of indices, the matrix times vector operation is given by the general formula for the $i^{th}$ row of the output vector, where $i=1,\dots ,n$:

$$(C\mathbf{w}{)}_i=\sum _{j=1}^m{C}_{ij}{w}_j.$$

Matrices acting on vectors (part 3)

The formulas for the action of a matrix on a vector should be understood in two different ways.

• For concrete calculations, it is best to think of each row of the output vector as the dot product of the corresponding matrix row with the input vector.
• For conceptual and abstract understanding, it is better to think of the operation as sending each standard basis vector to the corresponding column vector of the matrix.

Let’s consider now the second way in more detail. A matrix can be considered as a list of column vectors. For example:

$$A=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)=\left(\begin{array}{cc}{\mathbf{a}}_1& {\mathbf{a}}_2\end{array}\right),\text{ with }{\mathbf{a}}_1=\left(\begin{array}{c}{a}_{11}\\ a_{21}\end{array}\right),{\mathbf{a}}_2=\left(\begin{array}{c}{a}_{12}\\ a_{22}\end{array}\right).$$

Then $A\mathbf{u}=u_1{\mathbf{a}}_1+u_2{\mathbf{a}}_2$:

$$\left(\begin{array}{cc}{\mathbf{a}}_1& {\mathbf{a}}_2\end{array}\right)\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right)=u_1\left(\begin{array}{c}{a}_{11}\\ a_{21}\end{array}\right)+u_2\left(\begin{array}{c}{a}_{12}\\ a_{22}\end{array}\right).$$

Recall the standard basis vectors are ${\mathbf{e}}_1=\left(\begin{array}{c}1\\ 0\end{array}\right)$ and ${\mathbf{e}}_2=\left(\begin{array}{c}0\\ 1\end{array}\right)$. So:

$$A{\mathbf{e}}_1=\left(\begin{array}{c}{a}_{11}\\ a_{21}\end{array}\right),A{\mathbf{e}}_2=\left(\begin{array}{c}{a}_{12}\\ a_{22}\end{array}\right).$$

Similarly, in general:

$$C{\mathbf{e}}_i={\mathbf{c}}_i=\left(\begin{array}{c}{c}_{1i}\\ c_{2i}\\ ⋮\\ c_{ni}\end{array}\right).$$

Diagonal matrices - stretching

The main diagonal of a matrix $A=(a_{ij})$ consists of the entries $a_{11},a_{22},\dots ,a_{nn}$. A matrix is called diagonal when all entries are zero except on the main diagonal. The action of a diagonal matrix is to stretch each spatial dimension according to the corresponding diagonal number:

$$\left(\begin{array}{ccc}{a}_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right)\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=\left(\begin{array}{c}{a}_{1}x\\ a_{2}y\\ a_{3}z\end{array}\right).$$

This action is very easy to express algebraically in terms of standard basis vectors:

$$A{\mathbf{e}}_1=a_1{\mathbf{e}}_1,A{\mathbf{e}}_2=a_2{\mathbf{e}}_2,A{\mathbf{e}}_3=a_3{\mathbf{e}}_3.$$

In other words, standard basis vectors are sent to scalar multiples of themselves. This means that standard basis vectors are eigenvectors (“self-vectors”) for a diagonal matrix, and the diagonal numbers (the scalar multiples) are eigenvalues (“self-values”).

When the diagonal entries are all the same, the effect is to scale all of space by equal proportions:

$$\left(\begin{array}{ccc}a& 0& 0\\ 0& a& 0\\ 0& 0& a\end{array}\right)\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=\left(\begin{array}{c}ax\\ ay\\ az\end{array}\right)=a\left(\begin{array}{c}x\\ y\\ z\end{array}\right).$$

In this case, when $a>1$ the transformation is called a dilation because vectors expand away from the origin, and when $a<1$ it is called a contraction because vectors shrink towards the origin.

Shearing

A simple shear mapping (also shear transformation or transvection) performs a displacement of one coordinate by a proportion of another. The matrix of a 2D shear in $x$ is given by:

$$\left(\begin{array}{cc}1& m\\ 0& 1\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}x+my\\ y\end{array}\right).$$

Horizontal lines remain horizontal, while vertical lines take on a new slope of $1/m$. Interestingly, shearing preserves area. (We will see a very simple explanation later.)

There is also shear in $y$:

$$\left(\begin{array}{cc}1& 0\\ m& 1\end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}x\\ y+mx\end{array}\right).$$

A shear in 3D might look like this:

$$\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& m\\ 0& 0& 1\end{array}\right)\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=\left(\begin{array}{c}x\\ y+mz\\ z\end{array}\right).$$

The shear takes place in the component corresponding to the row of the matrix with $m$.
There is a general definition of shear mapping in $n$D space, but we do not use it.

Rotating

A rotation in 2D by the angle $\theta$ (counterclockwise) can be performed by a matrix:

$$\left(\begin{array}{cc}\cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{array}\right)\left(\begin{array}{c}x\\ y\end{array}\right)=\left(\begin{array}{c}x\cos \theta -y\sin \theta \\ x\sin \theta +y\cos \theta \end{array}\right).$$

Rotating (part 2)

From the matrix for rotation by $\theta$ we can deduce that counterclockwise rotation by $90^{\circ }$ is given in matrix form by $\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)$, which acts by $(x,y)\mapsto (-y,x)$, and clockwise rotation by $90^{\circ }$ is given in matrix form by $\left(\begin{array}{cc}0& 1\\ -1& 0\end{array}\right)$, which acts by $(x,y)\mapsto (y,-x)$. Rotation by $180^{\circ }$ is given by $\left(\begin{array}{cc}-1& 0\\ 0& -1\end{array}\right)$, which acts by $(x,y)\mapsto (-x,-y)$, and this is simply reflection through the origin.

Study of 3D rotations has myriad applications in science and engineering, but rotations in 3D are much more complicated than in 2D, and we do not investigate the theory very much here. Some basic rotations are given by simple 2D rotations in the separate pairs of coordinates:

$$R_{xy}(\theta )=\left(\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0& 0& 1\end{array}\right),R_{xz}(\theta )=\left(\begin{array}{ccc}\cos \theta & 0& -\sin \theta \\ 0& 1& 0\\ \sin \theta & 0& \cos \theta \end{array}\right),R_{yz}(\theta )=\left(\begin{array}{ccc}1& 0& 0\\ 0& \cos \theta & -\sin \theta \\ 0& \sin \theta & \cos \theta \end{array}\right)$$

These rotations are called yaw, pitch, and roll, respectively. (These terms apply to an airplane flying in the $xy$-plane in the $+x$ axis direction.) Simple rotations like these in two chosen axes are also called Givens rotations (after J. Wallace Givens). They are easily generalized to $n$D space.

Projecting

A projection matrix acts on vectors by sending them to the nearest point in some chosen linear subspace, for example on a line (in 2D space) or on a plane (in 3D space). The chosen subspace has lower dimension than all of space, and the image under projection is like a shadow of the original.

Consider the projection of $\mathbf{u}$ to the direction of $\mathbf{v}$. Let us assume that $\mathbf{v}$ is a unit vector; then the projection formula simplifies to:

$${\mathrm{proj}}_{\mathbf{v}}(\mathbf{u})=\mathbf{v}(\mathbf{v}\cdot \mathbf{u}).$$

We can express this formula using a matrix. In 2D we have:

$$(\mathbf{v}\cdot \mathbf{u})=\left(\begin{array}{cc}{v}_1& v_2\end{array}\right)\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right)=v_1{u}_1+v_2{u}_2,$$

and

$$\mathbf{v}(\mathbf{v}\cdot \mathbf{u})=\left(\begin{array}{c}{v}_1(\mathbf{v}\cdot \mathbf{u})\\ v_2(\mathbf{v}\cdot \mathbf{u})\end{array}\right)=\left(\begin{array}{c}{v}_1(v_1{u}_1+v_2{u}_2)\\ v_2(v_1{u}_1+v_2{u}_2)\end{array}\right)=\left(\begin{array}{c}{v}_1{v}_1{u}_1+v_1{v}_2{u}_2\\ v_2{v}_1{u}_1+v_2{v}_2{u}_2\end{array}\right)=\left(\begin{array}{cc}{v}_1{v}_1& v_1{v}_2\\ v_2{v}_1& v_2{v}_2\end{array}\right)\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right).$$

The matrix $\left(\begin{array}{cc}{v}_1{v}_1& v_1{v}_2\\ v_2{v}_1& v_2{v}_2\end{array}\right)$ actually has a name: it is called the outer product $\mathbf{v}{\mathbf{v}}^{\top }$, where ${\mathbf{v}}^{\top }$ is the matrix $(v_1{v}_2)$. In more detail: ${\mathbf{v}}^{\top }$ is a $1\times 2$ row vector matrix which, when applied to any vector $\mathbf{u}$, gives the scalar quantity $\mathbf{v}\cdot \mathbf{u}$. This horizontal matrix can be embedded inside each row of the vector $\mathbf{v}$ by multiplying ${\mathbf{v}}^{\top }$ by the component in that row of $\mathbf{v}$ and substituting the result into the row, thereby converting the vector $\mathbf{v}$ into a matrix. The action of this matrix on $\mathbf{u}$ is first to take the dot product $\mathbf{v}\cdot \mathbf{u}$, and then to apply this scalar to each row of $\mathbf{v}$, in other words multiplying $\mathbf{v}$ overall by this scalar.

In conclusion, the operation ${\mathrm{proj}}_{\mathbf{v}}(-)$ is a linear mapping, and it can be represented as the action of a matrix:

$$\left[{\mathrm{proj}}_{\mathbf{v}}\right]=\left(\begin{array}{cc}{v}_1{v}_1& v_1{v}_2\\ v_2{v}_1& v_2{v}_2\end{array}\right).$$

Now if $\mathbf{v}$ is not a unit vector, this matrix is incorrect. The matrix can be corrected by multiplying every coefficient by $(\mathbf{v}\cdot \mathbf{v}{)}^{-1}$, which is $1/(v_1{v}_1+v_2{v}_2)$.

Projecting (part 2)

The same matrix formula can be extended to 3D and $n$D. In 3D the matrix would be:

$$\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\end{array}\right)\left(\begin{array}{ccc}{v}_1& v_2& v_3\end{array}\right)=\left(\begin{array}{ccc}{v}_1{v}_1& v_1{v}_2& v_1{v}_3\\ v_2{v}_1& v_2{v}_2& v_2{v}_3\\ v_3{v}_1& v_3{v}_2& v_3{v}_3\end{array}\right).$$

For $\mathbf{v}$ a unit vector, this matrix acts on $\mathbf{u}$ by giving ${\mathrm{proj}}_{\mathbf{v}}(\mathbf{u})$, the projection to the direction of $\mathbf{v}$. We can see this because it clearly gives $\mathbf{v}(\mathbf{v}\cdot \mathbf{u})$.

It is good to highlight the matrices of projection to the standard basis vectors:

$$\left[{\mathrm{proj}}_{{\mathbf{e}}_1}\right]=\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right),\left[{\mathrm{proj}}_{{\mathbf{e}}_2}\right]=\left(\begin{array}{cc}0& 0\\ 0& 1\end{array}\right).$$

Reflecting

A reflection matrix is a matrix that acts on vectors by reflecting them across a hyperplane that passes through the origin. These operations are called Householder reflections.

(A hyperplane is a plane of one dimension lower than the surrounding space. So, a hyperplane in 2D is a line, and a hyperplane in 3D is a 2D plane. A hyperplane in $n$D would be an $(n-1)$D plane.)

The matrix of a reflection can be derived from the matrix of a certain projection by considering a formula for the reflection in terms of dot products.

If $\mathbf{u}$ is any vector, the reflection of $\mathbf{u}$ in the hyperplane through the origin, perpendicular to the vector $\mathbf{v}$ is given by:

$${\mathrm{refl}}_{\mathbf{v}}(\mathbf{u})=\mathbf{u}-2{\mathrm{proj}}_{\mathbf{v}}(\mathbf{u}).$$

To see how the formula works, consider that $\mathbf{u}-{\mathrm{proj}}_{\mathbf{v}}(\mathbf{u})={\mathbf{u}}_{\perp }$ is the component of $\mathbf{u}$ perpendicular to $\mathbf{v}$, meaning that it lies in the hyperplane of reflection. Adding ${\mathbf{u}}_{\parallel }$ to ${\mathbf{u}}_{\perp }$ recovers $\mathbf{u}$ by bringing back the parallel part on the same side of the hyperplane as $\mathbf{v}$. Subtracting the term ${\mathbf{u}}_{\parallel }$ a second time does the opposite, it restores the part parallel to $\mathbf{v}$ but on the other side of the hyperplane.

Reasoning via the hyperplane is not strictly necessary in 2D, but the same reasoning also works in $n$D. In 3D the same formula gives the reflection across the plane through the origin, perpendicular to the vector $\mathbf{v}$. Here, ${\mathrm{proj}}_{\mathbf{v}}(\mathbf{u})$ gives the component of $\mathbf{u}$ coming out of this plane in along $\mathbf{v}$. Subtracting once gives a vector inside the plane; subtracting twice gives the mirror image vector on the opposite side of the plane.

A 2D reflection in the hyperplane perpendicular to $\mathbf{v}$ can be represented as the action of a matrix:

$$\begin{array}{rl}\left[{\mathrm{refl}}_{\mathbf{v}}\right]& =\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)-2\left[{\mathrm{proj}}_{\mathbf{v}}\right]\\ & =\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)-2\left(\begin{array}{cc}{v}_1{v}_1& v_1{v}_2\\ v_2{v}_1& v_2{v}_2\end{array}\right)\\ & =\left(\begin{array}{cc}1-2v_1{v}_1& -2v_1{v}_2\\ -2v_2{v}_1& 1-2v_2{v}_2\end{array}\right).\end{array}$$

The action of this matrix sends a vector $\mathbf{u}$ to the vector $\mathbf{u}-2{\mathrm{proj}}_{\mathbf{v}}(\mathbf{u})$. (Check this fact yourself!)

Permuting

A permutation is a way of reassigning some labels to a collection of items. For example, if some marbles numbered $1,\dots ,n$ are placed (out of order) in bins numbered $1,\dots ,n$, then a permutation corresponds to withdrawing the marbles and placing them in the same bins but in a new order.

A permutation matrix is a matrix in which each column and each row has a single $1$, and all other entries are $0$. For example:

$$\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right),\left(\begin{array}{ccc}0& 1& 0\\ 1& 0& 0\\ 0& 0& 1\end{array}\right),\left(\begin{array}{cccc}0& 0& 0& 1\\ 1& 0& 0& 0\\ 0& 0& 1& 0\\ 0& 1& 0& 0\end{array}\right).$$

The action of a permutation matrix on a vector is simply to permute the rows of the vector:

$$\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\left(\begin{array}{c}{v}_1\\ v_2\end{array}\right)=\left(\begin{array}{c}{v}_2\\ v_1\end{array}\right),\left(\begin{array}{ccc}0& 1& 0\\ 1& 0& 0\\ 0& 0& 1\end{array}\right)\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\end{array}\right)=\left(\begin{array}{c}{v}_2\\ v_1\\ v_3\end{array}\right),$$ $$\left(\begin{array}{cccc}0& 0& 0& 1\\ 1& 0& 0& 0\\ 0& 0& 1& 0\\ 0& 1& 0& 0\end{array}\right)\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\\ v_4\end{array}\right)=\left(\begin{array}{c}{v}_4\\ v_1\\ v_3\\ v_2\end{array}\right).$$

Such a matrix acts by permuting rows because each row, with its single entry of $1$, “picks out” the entry of the vector whose row corresponds to the column of this $1$ (all others are given times $0$); and since each column of the matrix has only a single $1$, each row of the vector is “picked out” only one time.

The action of a permutation matrix on standard basis vectors is easy to describe: it permutes them.