Theory

Matrix multiplication

Suppose we have a matrix $A$ and a vector $u$ that $A$ acts upon. We have written “$Au$” for the output vector given by letting $A$ act on $u$.

Matrices act by sending vectors to other vectors. It is useful to consider another matrix that acts upon the output vector. For example, if $\mathbf{v}:=A\mathbf{u}$, and if $B$ is a matrix that acts upon $v$, we can write $B(A\mathbf{u})=B\mathbf{v}$. (The colon in $\mathbf{v}:=A\mathbf{u}$ means “$\mathbf{v}$ is defined as $A\mathbf{u}$.”)

Suppose we define $\mathbf{w}:=B\mathbf{v}$, so that $\mathbf{w}=B(A\mathbf{u})$. Then we say that $w$ is the output of $\mathbf{u}$ under a composition of actions of the matrices $A$ and $B$. It turns out that if we start with two matrices $A$ and $B$, the composition action (letting the matrices act consecutively) can itself be represented by a single matrix. (The composition of linear transformations is a linear transformation.) Let us find this matrix in the case when $A$ and $B$ are $2\times 2$ matrices.

Suppose the entries of $A$ and $B$ are given by

$$A=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right),B=\left(\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right).$$

Now compute the action of $A$ followed by the action of $B$ on the vector $\mathbf{u}=\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right)$:

$$\begin{array}{rl}B(A\mathbf{u})& =\left(\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right)\left(\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right)\right)\\ & =\left(\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right)\left(\begin{array}{c}{a}_{11}{u}_1+a_{12}{u}_2\\ a_{21}{u}_1+a_{22}{u}_2\end{array}\right)\\ & =\left(\begin{array}{c}{b}_{11}(a_{11}{u}_1+a_{12}{u}_2)+b_{12}(a_{21}{u}_1+a_{22}{u}_2)\\ b_{21}(a_{11}{u}_1+a_{12}{u}_2)+b_{22}(a_{21}{u}_1+a_{22}{u}_2)\end{array}\right)\\ & =\left(\begin{array}{c}(b_{11}{a}_{11}+b_{12}{a}_{21})u_1+(b_{11}{a}_{12}+b_{12}{a}_{22})u_2\\ (b_{21}{a}_{11}+b_{22}{a}_{21})u_1+(b_{21}{a}_{12}+b_{22}{a}_{22})u_2\end{array}\right)\\ & =\left(\begin{array}{cc}{b}_{11}{a}_{11}+b_{12}{a}_{21}& b_{11}{a}_{12}+b_{12}{a}_{22}\\ b_{21}{a}_{11}+b_{22}{a}_{21}& b_{21}{a}_{12}+b_{22}{a}_{22}\end{array}\right)\left(\begin{array}{c}{u}_1\\ u_2\end{array}\right).\end{array}$$

Therefore, we define the product matrix $BA$ by this formula:

$$BA=\left(\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right)\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)=\left(\begin{array}{cc}{b}_{11}{a}_{11}+b_{12}{a}_{21}& b_{11}{a}_{12}+b_{12}{a}_{22}\\ b_{21}{a}_{11}+b_{22}{a}_{21}& b_{21}{a}_{12}+b_{22}{a}_{22}\end{array}\right).$$

With this definition and the above reasoning, we have that $B(A\mathbf{u})=(BA)\mathbf{u}$.
(Because of this equality, we can omit the parentheses and write simply $BA\mathbf{u}$, since the result is the same whether you act first by $A$ and then by $B$ or instead you first multiply the matrices and then act by $BA$.)

Just as for the formula for a matrix acting on a vector, we have multiple ways to interpret the formula for the matrix $BA$.

• Each entry of the product matrix $BA$ is given by taking the dot product of a row of $B$ with a column of $A$.
• Each column of the product matrix $BA$ is given by letting $B$ act upon the corresponding column vector of $A$. In other words, $BA=\left(\begin{array}{cc}B{\mathbf{a}}_1& B{\mathbf{a}}_2\end{array}\right)$. (Recall that the action of $B$ upon ${\mathbf{a}}_i$ has two interpretations of its own!)
• Each row of the product matrix $BA$ is given by letting the entries in the corresponding row vector of $B$ provide the coefficients for a linear combination of the rows of $A$.
  The first of these interpretations is best for doing calculations, and the second is best for understanding abstract theory.

Bottom-Right Method for computing $AB$:

E.g.:

Max Born, Nobel Prize lecture:

Matrix multiplication (part 2)

Identity matrix
Notice that the diagonal matrix with $1$’s on the diagonal acts upon another vector or matrix by doing nothing:

$$\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right)\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right)=\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right).$$

This matrix is called the identity matrix, since it preserves the identity of whatever it acts upon. It is frequently written “$I$”, regardless of the size of the matrix. So $I\mathbf{u}=\mathbf{u}$ for any $\mathbf{u}$.

Sometimes $I$ is written more specifically as $I_n$, where $n$ is the number of rows or columns. (Identity matrices always have the same number of rows and columns.)

Matrices acting on matrices

It is useful to think of matrices as acting upon other matrices by multiplying on the left. The effect is the same as the effect of acting on a vector, but it performs this (same) action on each column vector separately. The action is therefore the same across any given row.

Diagonal matrices
When two diagonal matrices are multiplied, the result is a diagonal matrix. The diagonal entries of the product are simply the products of the diagonal entries:

$$\left(\begin{array}{ccc}{a}_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right)\left(\begin{array}{ccc}{b}_1& 0& 0\\ 0& b_2& 0\\ 0& 0& b_3\end{array}\right)=\left(\begin{array}{ccc}{a}_1{b}_1& 0& 0\\ 0& a_2{b}_2& 0\\ 0& 0& a_3{b}_3\end{array}\right).$$

A consequence is that diagonal matrices commute with each other: $AB=BA$ when $A,B$ are both diagonal. (As seen with 3D rotations, frequently this is not true for matrix multiplication.)

Row-scale matrices
The diagonal matrices

$$A_1(\lambda )=\left(\begin{array}{ccc}\lambda & 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right),A_2(\lambda )=\left(\begin{array}{ccc}1& 0& 0\\ 0& \lambda & 0\\ 0& 0& 1\end{array}\right),A_3(\lambda )=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& \lambda \end{array}\right)$$

have the effect of scaling rows $1,2,3$ (respectively) by the number $\lambda$, without changing other rows, when they are multiplied on a vector or a matrix on the left. These matrices are called row-scale matrices.

Permutation matrices
When two permutation matrices are multiplied, the result is a permutation matrix. For example:

$$\left(\begin{array}{ccc}0& 1& 0\\ 1& 0& 0\\ 0& 0& 1\end{array}\right)\left(\begin{array}{ccc}1& 0& 0\\ 0& 0& 1\\ 0& 1& 0\end{array}\right)=\left(\begin{array}{ccc}0& 0& 1\\ 1& 0& 0\\ 0& 1& 0\end{array}\right).$$

Matrices acting on matrices (part 2)

Row-swap matrices
A permutation matrix acts upon another matrix (multiplying on the left) by permuting the rows of the matrix, in the same way that it would permute the rows of a vector:

$$\left(\begin{array}{cc}0& 1\\ 1& 0\end{array}\right)\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)=\left(\begin{array}{cc}{a}_{21}& a_{22}\\ a_{11}& a_{12}\end{array}\right).$$

The most important permutation type is one which simply swaps two rows and leaves the others unchanged. This type of matrix is called a row-swap matrix.

The row-swap action is performed by a permutation matrix which has $1$’s on the diagonal except for two rows. In these two rows, the ones from a diagonal matrix are swapped with each other.

Matrices acting on matrices (part 3)

Row-add matrices
A matrix that has $1$’s on the diagonal and the other entries $0$’s, except for a single additional entry $\lambda$, is called an elementary shear matrix, or a row-add matrix:

$$\left(\begin{array}{ccc}1& \lambda & 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right),\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ \lambda & 0& 1\end{array}\right),\text{etc.}$$

If the $\lambda$ entry is in the $i^{th}$ row and $j^{th}$ column, this matrix acts by taking the $j^{th}$ row (of a matrix or vector that it’s acting upon) and adding it into the $i^{th}$ row with a $\lambda$ coefficient:

$$\left(\begin{array}{ccc}1& \lambda & 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right)\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right)=\left(\begin{array}{ccc}{a}_{11}+\lambda a_{21}& a_{12}+\lambda a_{22}& a_{13}+\lambda a_{23}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right).$$

Remember that when a matrix is multiplied on the left on a vector or a matrix, the top row of the matrix controls the top row of the output.

Matrix algebra

Matrix addition and scaling
Matrices can be added, subtracted, and multiplied by scalers, just like vectors. These operations are done componentwise:

$$\begin{array}{rl}\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)+\left(\begin{array}{cc}{b}_{11}& b_{12}\\ b_{21}& b_{22}\end{array}\right)& =\left(\begin{array}{cc}{a}_{11}+b_{11}& a_{12}+b_{12}\\ a_{21}+b_{21}& a_{22}+b_{22}\end{array}\right),\\ \lambda \left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)& =\left(\begin{array}{cc}\lambda a_{11}& \lambda a_{12}\\ \lambda a_{21}& \lambda a_{22}\end{array}\right).\end{array}$$

In terms of the actions performed by these matrices, addition is like addition of functions: the action of a sum of matrices is to give the sum of the outputs under the actions of the separate matrices. For functions, this would be written $(f+g)(x)=f(x)+g(x)$. For matrices, we write $(A+B)\mathbf{u}=A\mathbf{u}+B\mathbf{u}$, and similarly for subtraction, scalar multiples, and other variations on basic algebra operations.

Matrix distributivity
It is important to notice that matrix multiplication is compatible with matrix addition in the usual sense of distributivity:

$$A(B+C)=AB+AC.$$

One way to prove this formula is to show that it is true after both sides are applied to an arbitrary vector $\mathbf{u}$. (To complete the proof, let $\mathbf{u}$ take the specific value of each standard basis vector ${\mathbf{e}}_i$ in turn, and you find that column $i$ of the matrices on each side must agree.) To show the equality after multiplying both sides on an arbitrary $\mathbf{u}$, we need to check distributivity of matrix actions on vectors in general:

Matrix algebra (part 2)

Matrix division
For real numbers, division and multiplication are inverse processes, meaning that division “undoes” the action of multiplication, and vice versa.

Any number can divide another number provided it is not zero. For matrices, the situation is more complex, because “being zero” is more complex. Many matrices have non-zero entries, yet they “act as zero” upon certain other vectors. When this happens, the action cannot be “undone”, because you cannot divide by the zero action performed on those vectors.

Matrix algebra (part 3)

So, some matrices can be divisors. These are called invertible matrices. Others cannot, and these are called non-invertible matrices. A matrix $A$ is invertible when the inverse matrix $A^{-1}$ exists.

When it exists, an inverse matrix $A^{-1}$ acts by undoing multiplication by $A$. Concretely, this means:

$$A^{-1}A\mathbf{u}=\mathbf{u},A{A}^{-1}\mathbf{u}=\mathbf{u}\text{for any vector }\mathbf{u}.$$

Notice in the second formula that the action of $A$ can be undone before $A$ even acts. (Ordinary division also works like this!) Another way to look at it is that $A$ acts by undoing the effect of $A^{-1}$. (Ordinary division: multiplication by $5$ undoes the effect of multiplying by $1/5$.)

Recall the “identity matrix” $I$ which satisfies $I\mathbf{u}=\mathbf{u}$ for any $\mathbf{u}$. So we can write the above formula using matrices:

$$A^{-1}A=I,A{A}^{-1}=I.$$

Notice that we have to write both formulas because matrix multiplication is not commutative! (Ordinary division: $5\cdot (1/5)=1$ and $(1/5)\cdot 5=1$ are equivalent because $ab=ba$ for any numbers.)

Inverse of $2\times 2$ matrices
There is a general formula for the inverse of any $2\times 2$ matrix that should be memorized:

$$A=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right),A^{-1}=\frac{1}{ad-bc}\left(\begin{array}{cc}d& -b\\ -c& a\end{array}\right).$$

Matrix algebra (part 4)

The inverse of $A$ is useful for finding a vector $\mathbf{u}$ which is sent to a given vector $\mathbf{v}$ by the action of a given matrix $A$:

$$A\mathbf{u}=\mathbf{v}\text{if and only if}\mathbf{u}=A^{-1}\mathbf{v}.$$

If we know $A$ and $\mathbf{v}$, and we want $\mathbf{u}$, this method will give us $\mathbf{u}$.

Matrix algebra (part 5)

Determinant of $2\times 2$ matrices
The determinant of a matrix $A$, written $\det A$, is a single number that is associated to the matrix.

For a $2\times 2$ matrix, the determinant is given by the formula in the divisor of the formula for $A^{-1}$:

$$\det A=ad-bc\text{when}A=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right).$$

We will learn many things about determinants. The important thing for now is that $A^{-1}$ exists if and only if $\det A\ne 0$.

Optional: interpretation of $2\times 2$ determinant
Just for fun, here is a visual proof (by S. Golomb) that $\det A$ gives the area of the parallelogram spanned by the column vectors of $A$, namely $\left(\begin{array}{c}a\\ c\end{array}\right)$ and $\left(\begin{array}{c}b\\ d\end{array}\right)$:

Here is another visual for parallelograms from obtuse vectors:

Eigenvectors, eigenvalues

We saw that standard basis vectors ${\mathbf{e}}_i$ are eigenvectors for diagonal matrices, and the diagonal entries are the eigenvalues. If $A$ has diagonal entries $a_1,a_2,a_3$, then $A{\mathbf{e}}_i=a_i{\mathbf{e}}_i$.

Now consider the problem of finding eigenvectors and eigenvalues for a non-diagonal matrix $A$. This means finding a vector $\mathbf{v}\ne 0$ and a scalar $\lambda$ such that $A\mathbf{v}=\lambda \mathbf{v}$.

Suppose we start with a hypothetical $\lambda$. Write $\lambda I_2$ for the matrix $\left(\begin{array}{cc}\lambda & 0\\ 0& \lambda \end{array}\right)$. So we wish to solve $A\mathbf{v}=(\lambda I_2)\mathbf{v}$, which is equivalent to $(A-\lambda I_2)\mathbf{v}=0$. Now write $A_{\lambda }:=A-\lambda I_2$. So we wish to find $\mathbf{v}$ such that $A_{\lambda }\mathbf{v}=0$.

Notice that if $A_{\lambda }$ is invertible, then there is no solution. (The only solution would be $\mathbf{v}=0$ and that is not allowed for an eigenvector.) The reason is that if $A_{\lambda }^{-1}$ exists, we can let it act upon both sides of the equation $A_{\lambda }\mathbf{v}=0$ to obtain $\mathbf{v}=A_{\lambda }^{-1}0=0$. Therefore, we only have a chance of solving this if $A_{\lambda }$ is non-invertible. Recall from above that $A_{\lambda }$ is non-invertible precisely when $\det A_{\lambda }=0$.

It turns out we can solve the equation if $\det A_{\lambda }=0$. Let us do this by hand for $2\times 2$ matrices.

Derivation that if $\det M=0$ then $M\mathbf{v}=0$ can be solved with $\mathbf{v}\ne 0$

Let $M=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)$. Assume $\det M=0$, which means $ad-bc=0$ or $ad=bc$ or $c/a=d/b$. Write $x$ for the common ratio $c/a=d/b$. Thus $M=\left(\begin{array}{cc}a& b\\ xa& xb\end{array}\right)$, and now we wish to solve the equation:

$$\left(\begin{array}{cc}a& b\\ xa& xb\end{array}\right)\left(\begin{array}{c}{v}_1\\ v_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right).$$

This means $a{v}_1+b{v}_2=0$ and $xa{v}_1+xb{v}_2=0$. Dividing the latter by $x$ we arrive at the former, so we can take any $(v_1,v_2)$ that satisfy $v_1/v_2=-b/a$. Let us choose $(v_1,v_2)=(-b,a)$. You can now check directly that it works:

$$\left(\begin{array}{cc}a& b\\ xa& xb\end{array}\right)\left(\begin{array}{c}-b\\ a\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right).$$

If $\det A_{\lambda }=0$, there must be a vector $\mathbf{v}\ne 0$ such that $A\mathbf{v}=\lambda \mathbf{v}$ for some scalar $\lambda$

Proof: Define $A=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)$, and so

$$A_{\lambda }=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)-\left(\begin{array}{cc}\lambda & 0\\ 0& \lambda \end{array}\right)=\left(\begin{array}{cc}a-\lambda & b\\ c& d-\lambda \end{array}\right).$$

Suppose $\det A_{\lambda }=0$. By the formula for $\det$, this means $(a-\lambda )(d-\lambda )-bc=0$. That is a quadratic equation in $\lambda$ and could be rewritten as ${\lambda }^2+(-a-d)\lambda +(ad-bc)=0$. By the quadratic formula, the solutions are $\lambda =\frac{1}{2}\left(a+d\pm \sqrt{(a-d{)}^2+4bc}\right)$. Write these two solutions as ${\lambda }_1$ and ${\lambda }_2$.

So we have found ${\lambda }_1$ and ${\lambda }_2$ with $\det A_{{\lambda }_1}=0$ and $\det A_{{\lambda }_2}=0$. In either case we can solve for a ${\mathbf{v}}_1\ne 0$ such that $A_{{\lambda }_1}{\mathbf{v}}_1=0$ and a ${\mathbf{v}}_2$ such that $A_{{\lambda }_1}{\mathbf{v}}_2=0$ using the method in the box above.

Summary

• Matrices can be multiplied by each other. Express $A=\left(\begin{array}{ccc}{\mathbf{a}}_1& {\mathbf{a}}_2& {\mathbf{a}}_3\end{array}\right)$ as a row of column vectors; then $BA$ is the matrix which is the row of column vectors $\left(\begin{array}{ccc}B{\mathbf{a}}_1& B{\mathbf{a}}_2& B{\mathbf{a}}_3\end{array}\right)$. In other words, $BA$ is computed by letting $B$ act upon the columns of $A$.
• Matrices acting on vectors satisfy $A(\mathbf{u}+\mathbf{v})=A\mathbf{u}+A\mathbf{v}$, and acting on matrices they satisfy $A(B+C)=AB+AC$.
• The identity matrix $I_3=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right)$ has the property that $I_{3}A=A$ and $A{I}_3=A$ for any other matrix $A$.
• Row-scale matrices act on a matrix (or vector) by scaling a chosen row. Row-swap matrices act by swapping two rows. Row-add matrices act by adding one row into another.
• An “invertible” matrix $A$ is one having an inverse $A^{-1}$, which has the property that $A^{-1}A=A{A}^{-1}=I_n$. (Here $A$ is an $n\times n$ matrix.) For a $2\times 2$ matrix, its inverse is given by a formula: $$A=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right),A^{-1}=\frac{1}{ad-bc}\left(\begin{array}{cc}d& -b\\ -c& a\end{array}\right).$$
• The inverse matrix $A^{-1}$ can be used to solve for $\mathbf{u}$ in the equation $A\mathbf{u}=\mathbf{v}$, where $A$ and $\mathbf{v}$ are both given and $\mathbf{u}$ is sought.
• The determinant $\det A$ for the $2\times 2$ matrix $A$ (as above) is the number $ad-bc$. The matrix $A$ is invertible if and only if $\det A\ne 0$.
• Given the matrix $A$, an eigenvector $\mathbf{v}$ is a vector ($\ne 0$) such that $A\mathbf{v}=\lambda \mathbf{v}$ for some scalar $\lambda$ called the eigenvalue corresponding to $\mathbf{v}$.
• We can find eigenvalues and eigenvectors by: setting $\det (A-\lambda I_2)=0$ and solving a quadratic equation to find ${\lambda }_1$, ${\lambda }_2$, then plugging these in for $\lambda$ and writing $A_{{\lambda }_1}=A-{\lambda }_1{I}_2$ and $A_{{\lambda }_1}=A-{\lambda }_1{I}_2$, and then manually solving for vectors ${\mathbf{v}}_1$ and ${\mathbf{v}}_2$ satisfying $A_{{\lambda }_1}{\mathbf{v}}_1=0$ and $A_{{\lambda }_2}{\mathbf{v}}_2=0$. This is possible using the fact that one row of $A_{{\lambda }_i}$ will be a multiple $x$ times the other row.