Theory

Introduction

In this section we study some interrelated concepts: orthogonal matrices, symmetric matrices, inner products, quadratic forms, and positive definite matrices.

Orthogonal matrices

A matrix $Q$ is called orthogonal when it satisfies $A^{\mathsf{T}}A=I_n$.

‘Orthogonal’ $=$ ‘orthonormal’ for matrices

The term orthonormal is also used for the same class of matrices. (Warning! Not so for vectors! Orthogonal vectors are merely perpendicular, while orthonormal vectors are also unit vectors.)

Now consider the meaning of the formula $Q^{\mathsf{T}}Q=I_n$. Remember the rule for matrix products (e.g. in $3$D):

$$BA=\left(\begin{array}{ccc}B{\mathbf{a}}_1& B{\mathbf{a}}_2& B{\mathbf{a}}_3\end{array}\right).$$

What is $Q^{\mathsf{T}}{\mathbf{q}}_i$? Each row of this vector is the dot product of the same row of $Q^{\mathsf{T}}$ with the vector ${\mathbf{q}}_i$. In turn, each row of $Q^{\mathsf{T}}$ is from a column of $Q$. Combining all these claims with the product rule, column $i$ of $Q^{\mathsf{T}}Q$ has rows equal to the dot products of the various columns of $Q$ with the specific column $i$ of $Q$. Specifically, $(Q^{\mathsf{T}}Q{)}_{ij}={\mathbf{q}}_j\cdot {\mathbf{q}}_i$. In conclusion, $Q^{\mathsf{T}}Q=I_n$ means that ${\mathbf{q}}_j\cdot {\mathbf{q}}_i=\{\begin{array}{ll}1& i=j\\ 0& i\ne j.\end{array}$

It follows that $Q^{\mathsf{T}}Q=I_n$ is another way of saying that ${\mathbf{q}}_j$ and ${\mathbf{q}}_i$ are mutually orthonormal. So, if you put an orthonormal basis into the column vectors of a matrix, the resulting matrix is orthogonal. Conversely, if a matrix is orthogonal, then its columns form an orthonormal basis. (They are orthonormal by the definition of orthogonal, and they are a basis because they are perpendicular and thus independent.)

An important fact about orthogonal matrices is that they correspond geometrically to rotations and reflections (and combinations of these actions called ‘rotoreflections’).

Orthogonal matrices preserve the dot product, preserving lengths and angles

The action of an orthogonal matrix preserves the dot product:

$$(Q\mathbf{u})\cdot (Q\mathbf{v})=\mathbf{u}\cdot \mathbf{v}.$$

Why $(Q\mathbf{u})\cdot (Q\mathbf{v})=\mathbf{u}\cdot \mathbf{v}$

To see this, recall (1) our interpretation of row vectors as $1\times n$ matrices, and (2) the dot product with $\mathbf{x}$ as the action of the transpose row vector ${\mathbf{x}}^{\mathsf{T}}$ acting by matrix multiplication on the left, so e.g. $\mathbf{x}\cdot \mathbf{w}={\mathbf{x}}^{\mathsf{T}}\mathbf{w}$. Using this, we derive:

$$\begin{array}{rl}(Q\mathbf{u})\cdot (Q\mathbf{v})& =(Q\mathbf{u}{)}^{\mathsf{T}}(Q\mathbf{v})\\ & =({\mathbf{u}}^{\mathsf{T}}{Q}^{\mathsf{T}})(Q\mathbf{v})\\ & ={\mathbf{u}}^{\mathsf{T}}(Q^{\mathsf{T}}Q)\mathbf{v}\\ & ={\mathbf{u}}^{\mathsf{T}}(I_n)\mathbf{v}\\ & ={\mathbf{u}}^{\mathsf{T}}\mathbf{v}\\ & =\mathbf{u}\cdot \mathbf{v}.\end{array}$$

Symmetric matrices

A symmetric matrix $A$ is a matrix that satisfies $A^{\mathsf{T}}=A$. In terms of coefficients, this means it satisfies $a_{ij}=a_{ji}$. A symmetric matrix is symmetric about reflection in a mirror that runs down the main diagonal:

$$\text{Non-symmetric:}\left(\begin{array}{cc}1& 2\\ 3& 4\end{array}\right),\left(\begin{array}{cc}1& 2\\ -2& 1\end{array}\right)$$ $$\text{Symmetric:}\left(\begin{array}{cc}1& 3\\ 3& -2\end{array}\right),\text{antisymmetric:}\left(\begin{array}{cc}0& 3\\ -3& 0\end{array}\right)$$

An antisymmetric matrix (also called skew-symmetric) satisfies $A^{\mathsf{T}}=-A$, in other words $a_{ij}=-a_{ji}$.

Symmetric matrices do arise naturally in many applications. This can happen for a variety of reasons. One example is PCA (principal component analysis): here the matrix in question arises as a “covariance matrix,” and it is symmetric because covariance is symmetric: $\mathrm{Cov}(X,Y)=\mathrm{Cov}(Y,X)$. Antisymmetric matrices are less common.

Symmetric matrices have some very nice properties:

Spectral theorem

Let $A$ be any $n\times n$ symmetric matrix, so $A^{\mathsf{T}}=A$. Then:

• (a) $A$ has the max number of $n$ eigenvalues (some might be repeated, but they still exist).
• (b) All eigenvalues of $A$ are real numbers.
• (c) Eigenvectors with distinct eigenvalues are perpendicular to each other.
• (d) There is an orthonormal basis $\mathcal{Q}=\{{\mathbf{q}}_1,\dots ,{\mathbf{q}}_n\}$ such that $A$ becomes diagonal in this basis: $$D=Q^{-1}AQ,\text{ where }Q=\left(\begin{array}{ccc}{\mathbf{q}}_1& \cdots & {\mathbf{q}}_n\end{array}\right)={}_{\mathcal{B}}{T}_{\mathcal{Q}}$$ in other words $A=QD{Q}^{-1}$. Notice that $Q$ is an orthogonal matrix, and ${\mathbf{q}}_i$ are eigenvectors of $A$.

Proofs of most of these facts are just a little beyond the scope of a first applied course. Part (c) is not hard though:

Orthogonal eigenvectors of a symmetric matrix

Suppose $A{\mathbf{q}}_1={\lambda }_1{\mathbf{q}}_1$ and $A{\mathbf{q}}_2={\lambda }_2{\mathbf{q}}_2$ and ${\lambda }_1\ne {\lambda }_2$. Assume, as we do for eigenvectors, that ${\lambda }_1\ne 0$. Then consider the sequence:

$$\begin{array}{rl}{\mathbf{q}}_1\cdot {\mathbf{q}}_2& ={\lambda }_1^{-1}({\lambda }_1{\mathbf{q}}_1\cdot {\mathbf{q}}_2)\\ & ={\lambda }_1^{-1}(A{\mathbf{q}}_1)\cdot {\mathbf{q}}_2\\ & ={\lambda }_1^{-1}(A{\mathbf{q}}_1{)}^{\mathsf{T}}{\mathbf{q}}_2\\ & ={\lambda }_1^{-1}({\mathbf{q}}_1^{\mathsf{T}}{A}^{\mathsf{T}}){\mathbf{q}}_2\\ & ={\lambda }_1^{-1}{\mathbf{q}}_1^{\mathsf{T}}(A{\mathbf{q}}_2)\\ & ={\lambda }_1^{-1}{\mathbf{q}}_1^{\mathsf{T}}({\lambda }_2{\mathbf{q}}_2)\\ & =({\lambda }_1^{-1}{\lambda }_2){\mathbf{q}}_1^{\mathsf{T}}{\mathbf{q}}_2\\ & =({\lambda }_1^{-1}{\lambda }_2){\mathbf{q}}_1\cdot {\mathbf{q}}_2.\end{array}$$

Therefore, we find that ${\mathbf{q}}_1\cdot {\mathbf{q}}_2=({\lambda }_1^{-1}{\lambda }_2){\mathbf{q}}_1\cdot {\mathbf{q}}_2$. This can only happen if ${\lambda }_1={\lambda }_2$, which we have denied by hypothesis, or else if ${\mathbf{q}}_1\cdot {\mathbf{q}}_2=0$, which must therefore be true.

The equation $A=QD{Q}^{\mathsf{T}}$ is equivalent to a very nice formula for $A$ called the spectral decomposition formula:

$$A={\lambda }_1({\mathbf{q}}_1{\mathbf{q}}_1^{\mathsf{T}})+{\lambda }_2({\mathbf{q}}_2{\mathbf{q}}_2^{\mathsf{T}})+\cdots +{\lambda }_n({\mathbf{q}}_n{\mathbf{q}}_n^{\mathsf{T}}).$$

In this formula, the ${\mathbf{q}}_i$ are the orthogonal basis given by the spectral theorem (remember they are also eigenvectors), and the ${\lambda }_i$ are the corresponding eigenvalues.

SVD prelude

The spectral decomposition is an instance of the singular value decomposition (SVD) which works for general matrices. Stay tuned for more in the section on the SVD!

A great thing about this formula is that each ${\mathbf{q}}_i{\mathbf{q}}_i^{\mathsf{T}}$ is actually a projection matrix onto ${\mathbf{q}}_i$:

$${\mathbf{q}}_i{\mathbf{q}}_i^{\mathsf{T}}=[{\mathrm{proj}}_{{\mathbf{q}}_i}].$$

Why? Of course ${\mathbf{q}}_i{\mathbf{q}}_i^{\mathsf{T}}\mathbf{v}={\mathbf{q}}_i({\mathbf{q}}_i\cdot \mathbf{v})$. Since ${\mathbf{q}}_i$ are unit vectors, we have ${\mathrm{proj}}_{{\mathbf{q}}_i}(\mathbf{v})=(\mathbf{v}\cdot {\mathbf{q}}_i){\mathbf{q}}_i$. This is the same thing, just written with a different order for the scalar factor (result of the dot product).

[Optional] Explaining the spectral decomposition formula

The spectral decomposition formula itself comes from writing out the matrices in $A=QD{Q}^{\mathsf{T}}$:

$$\begin{array}{rl}A=QD{Q}^{\mathsf{T}}& =\left(\begin{array}{ccc}{\mathbf{q}}_1& \cdots & {\mathbf{q}}_n\end{array}\right)\left(\begin{array}{ccc}{\lambda }_1& & 0\\ & \ddots & \\ 0& & {\lambda }_n\end{array}\right)\left(\begin{array}{c}{\mathbf{q}}_1^{\mathsf{T}}\\ ⋮\\ {\mathbf{q}}_n^{\mathsf{T}}\end{array}\right)\\ & =\left(\begin{array}{ccc}{\mathbf{q}}_1& \cdots & {\mathbf{q}}_n\end{array}\right)\left(\begin{array}{c}{\lambda }_1{\mathbf{q}}_1^{\mathsf{T}}\\ ⋮\\ {\lambda }_n{\mathbf{q}}_n^{\mathsf{T}}\end{array}\right)\\ & =\left(\begin{array}{ccc}{\mathbf{q}}_1& \cdots & {\mathbf{q}}_n\end{array}\right)\left(\left(\begin{array}{c}{\lambda }_1{\mathbf{q}}_1^{\mathsf{T}}\\ 0\\ ⋮\\ 0\end{array}\right)+\left(\begin{array}{c}0\\ {\lambda }_2{\mathbf{q}}_2^{\mathsf{T}}\\ ⋮\\ 0\end{array}\right)+\left(\begin{array}{c}0\\ 0\\ ⋮\\ {\lambda }_n{\mathbf{q}}_n^{\mathsf{T}}\end{array}\right)\right)\\ & ={\mathbf{q}}_1{\lambda }_1{\mathbf{q}}_1^{\mathsf{T}}+\cdots +{\mathbf{q}}_n{\lambda }_n{\mathbf{q}}_n^{\mathsf{T}}\\ & ={\lambda }_1({\mathbf{q}}_1{\mathbf{q}}_1^{\mathsf{T}})+{\lambda }_2({\mathbf{q}}_2{\mathbf{q}}_2^{\mathsf{T}})+\cdots +{\lambda }_n({\mathbf{q}}_n{\mathbf{q}}_n^{\mathsf{T}}).\end{array}$$

Quadratic forms and symmetric matrices

A quadratic form is a function $Q:{\mathbb{R}}^n\to \mathbb{R}$ from vectors to scalars that can be written as a polynomial with all terms having degree $2$. For example:

$$\begin{array}{rl}& \text{In }2\mathrm{D}:Q(\left(\begin{array}{c}x\\ y\end{array}\right))=a{x}^2+b{y}^2+cxy\\ & \text{In }3\mathrm{D}:Q(\left(\begin{array}{c}x\\ y\\ z\end{array}\right))=a{x}^2+b{y}^2+c{z}^2+dxy+exz+fyz\end{array}$$

In short, a quadratic form is just a polynomial in several variables that is homogeneous of degree $2$. (‘Homogeneous’ just means that all terms have the same degree.) The difference between quadratic forms and quadratic polynomials is that the latter could also have terms of degree zero and degree one.

Applications of quadratic forms

The simplest possible quadratic form is the sum of squares: $Q(x,y,z)=x^2+y^2+z^2$. Notice that this form gives the square distance from $(x,y,z)$ to $(0,0,0)$. One strong application of linear algebra is the ability to extremize quadratric forms. For example, in Linear Least Squares, we extremize the distance from a point to a certain span. We will shortly see how to represent this distance as a quadratic form. In Weighted LLS, we minimize the distance with inequal weights applied to each term, such as in $Q(x,y,z)=4x^2+y^2+9z^2$. This kind of weighting is useful when some data is more reliable than other data, so it should be given more weight.

Quadratic forms and symmetric matrices (part 2)

Key fact:

Matrix of a quadratic form

Every quadratic form $Q$ can be represented uniquely by a symmetric matrix, meaning there is some symmetric $A$ such that $Q(\mathbf{x})={\mathbf{x}}^{\mathsf{T}}A\mathbf{x}$.

So there is a perfect correspondence between quadratic forms and symmetric matrices.

Quadratic forms and symmetric matrices (part 3)

A quadratic form can always be converted into a new quadratic form that has no cross terms by a suitable linear change of variables. If the original variables are the components of $\mathbf{x}$, then a linear change of variables would be a vector $\mathbf{y}$ defined by $\mathbf{y}={}_{{\mathcal{B}}^{\prime }}{T}_{\mathcal{B}}(\mathbf{x})$ for some change of basis transfer matrix ${}_{{\mathcal{B}}^{\prime }}{T}_{\mathcal{B}}$.

Principal axes theorem: eliminating cross terms

Given a quadratic form $Q(\mathbf{x})$ defined by the symmetric matrix $A$, suppose $A=PD{P}^{\mathsf{T}}$. Set $\mathbf{y}=P^{\mathsf{T}}\mathbf{x}$, so also $\mathbf{x}=P\mathbf{y}$. Then $Q(P\mathbf{y})$ has no cross terms in the $\mathbf{y}$ variables.

Derivation

First plug in the new variables and manipulate:

$${\mathbf{x}}^{\mathsf{T}}A\mathbf{x}=(P\mathbf{y}{)}^{\mathsf{T}}A(P\mathbf{y})={\mathbf{y}}^{\mathsf{T}}(P^{\mathsf{T}}AP)\mathbf{y}={\mathbf{y}}^{\mathsf{T}}D\mathbf{y}$$

This last expression has no cross terms:

$$\begin{array}{rl}{\mathbf{y}}^{\mathsf{T}}D\mathbf{y}& =\left(\begin{array}{ccc}{y}_1& \cdots & y_n\end{array}\right)\left(\begin{array}{ccc}{\lambda }_1& & 0\\ & \ddots & \\ 0& & {\lambda }_n\end{array}\right)\left(\begin{array}{c}{y}_1\\ ⋮\\ y_n\end{array}\right)\\ & \\ & ={\lambda }_1{y}_1^2+\cdots +{\lambda }_n{y}_n^2.\end{array}$$

Constrained optimization, min-max principle

One simple yet important application of elimination of cross terms is constrained optimization. The goal of constrained optimization is to find the extreme values of a quadratic form $Q(\mathbf{x})$ where $\mathbf{x}$ is constrained to be a unit vector, so ${\mathbf{x}}^{\mathsf{T}}\mathbf{x}=1$. Geometrically, this means: find the direction of fastest increase or direction of fastest decrease of $Q$, and find the rate of increase / decrease in those directions.

If we have written $Q(\mathbf{y})$ so as to eliminate cross terms:

$$Q(\mathbf{y})={\lambda }_1{y}_1^2+\cdots +{\lambda }_n{y}_n^2$$

then the max / min of $Q$ occurs for $\mathbf{y}={\mathbf{e}}_i$ where ${\lambda }_i$ is the max / min coefficient.

If $Q(\mathbf{x})$ is not written without cross terms, then we can first eliminate cross terms completely, and find the max / min among the set of ${\lambda }_i$. This will give the max / min value, and the input where this occurs in terms of the $y$ variables at $\mathbf{y}={\mathbf{e}}_i$. The change of variables must be made in reverse to find the corresponding $x$ input values where the max / min occurs. If the eigenvector ${\mathbf{q}}_i$ corresponding to ${\lambda }_i$ has already been found, its components are by definition the components of $P{\mathbf{e}}_i$ that we seek.

Constrained optimization, min-max principle (part 2)

However, it is not necessary to find all of the ${\lambda }_i$. These ${\lambda }_i$ are the eigenvalues / eigenvectors, and there are some methods of computing eigenvalues that locate the largest / smallest values by a numerical procedure that does not waste time finding the other values. The power method is an example of such a method. (Read more about the power method online, if you like.)

Min-max principle and Rayleigh quotient

A closely related topic is the min-max principle for finding eigenvalues using the Rayleigh quotient. Given any symmetric matrix $A$, its Rayleigh quotient is:

$$R(A,\mathbf{x})=\frac{{\mathbf{x}}^{\mathsf{T}}A\mathbf{x}}{{\mathbf{x}}^{\mathsf{T}}\mathbf{x}}.$$

Notice that the value of $R(A,\mathbf{x})$ is the same as the value of the quadratic form generated by $A$, evaluated at the unit vector given by first normalizing the input $\mathbf{x}$. Therefore, maximizing $R(A,\mathbf{x})$ over all possible $\mathbf{x}\ne 0$ is the same as maximizing the quadratic form of $A$ over all possible unit vectors.

The min-max principle is simply this: the min / max eigenvalues of $A$ are the min / max values of $R(A,\mathbf{x})$, and these occur when $\mathbf{x}$ is an eigenvector for that eigenvalue. The use of this principle is to find min / max eigenvalues / eigenvectors of a symmetric $A$ by extremizing $R(A,\mathbf{x})$ with numerical methods (like the power method and its variants).

The min-max principle and Rayleigh quotient is essentially just a notation and terminology for the process of normalizing $\mathbf{x}$ and plugging it into the quadratic form $Q$ arising from the symmetric matrix $A$.

Positive definite symmetric matrices

• A positive definite matrix is by definition a symmetric matrix all of whose eigenvalues are greater than zero.
• A negative definite matrix is by definition a symmetric matrix all of whose eigenvalues are less than zero.
• An indefinite matrix is by definition a symmetric matrix whose eigenvalues have mixed signs.

For example, in terms of spectral decompositions, we have:

$$\begin{array}{rl}A\text{ is positive definite:}A& =7{\mathbf{q}}_1{\mathbf{q}}_1^{\mathsf{T}}+{\mathbf{q}}_2{\mathbf{q}}_2^{\mathsf{T}}+3{\mathbf{q}}_3{\mathbf{q}}_3^{\mathsf{T}}\\ A\text{ is negative definite:}A& =-7{\mathbf{q}}_1{\mathbf{q}}_1^{\mathsf{T}}-{\mathbf{q}}_2{\mathbf{q}}_2^{\mathsf{T}}-3{\mathbf{q}}_3{\mathbf{q}}_3^{\mathsf{T}}\\ A\text{ is indefinite:}A& =7{\mathbf{q}}_1{\mathbf{q}}_1^{\mathsf{T}}-{\mathbf{q}}_2{\mathbf{q}}_2^{\mathsf{T}}+3{\mathbf{q}}_3{\mathbf{q}}_3^{\mathsf{T}}.\end{array}$$

Since the spectral decomposition reveals the quadratic form in the new variables, where ${\mathbf{q}}_i=P{\mathbf{e}}_i$, but these input variables cover the same set of possibilities, we also know:

Quadratic forms definiteness

• When $A$ is positive definite, then its form $Q(\mathbf{x})$ is always positive for $\mathbf{x}\ne 0$.
• When $A$ is negative definite, then its form $Q(\mathbf{x})$ is always negative for $\mathbf{x}\ne 0$.

Inner products and symmetric matrices

An inner product is a function sending two vector inputs to one scalar output:

$$⟨-,-⟩:(\mathbf{u},\mathbf{v})\mapsto ⟨\mathbf{u},\mathbf{v}⟩\in \mathbb{R},$$

which is linear in both inputs considered separately.

Rules of inner products:

• $⟨\mathbf{u},\mathbf{v}⟩=⟨\mathbf{v},\mathbf{u}⟩$ Symmetry.
• $⟨{\mathbf{u}}_1+{\mathbf{u}}_2,\mathbf{v}⟩=⟨{\mathbf{u}}_1,\mathbf{v}⟩+⟨{\mathbf{u}}_2,\mathbf{v}⟩$ Respects sums (also in $\mathbf{v}$ using symmetry).
• $⟨\lambda \mathbf{u},\mathbf{v}⟩=\lambda ⟨\mathbf{u},\mathbf{v}⟩$ Respects scalars (also in $\mathbf{v}$ using symmetry).
• $⟨\mathbf{u},\mathbf{u}⟩>0$ for all $\mathbf{u}\ne 0$, and $⟨\mathbf{u},\mathbf{u}⟩=0$ only occurs when $\mathbf{u}=0$.

We are very familiar with the dot product, which is a valid inner product:

$$\begin{array}{rl}⟨\mathbf{u},\mathbf{v}{⟩}_{\text{dot}}& =\mathbf{u}\cdot \mathbf{v}=\left(\begin{array}{c}{u}_1\\ u_2\\ u_3\end{array}\right)\cdot \left(\begin{array}{c}{v}_1\\ v_2\\ v_3\end{array}\right)\\ & ={\mathbf{u}}^{\mathsf{T}}\mathbf{v}=\left(\begin{array}{ccc}{u}_1& u_2& u_3\end{array}\right)\left(\begin{array}{c}{v}_1\\ v_2\\ v_3\end{array}\right)\\ & =u_1{v}_1+u_2{v}_2+u_3{v}_3.\end{array}$$

Using the linearity of matrix actions, we can create many candidate inner products by first acting by a symmetric matrix $A=A^{\mathsf{T}}$ and then taking the usual dot product:

$$⟨\mathbf{u},\mathbf{v}{⟩}_A=⟨\mathbf{u},A\mathbf{v}{⟩}_{\text{dot}}=\mathbf{u}\cdot A\mathbf{v}.$$

This definition of $⟨\mathbf{u},\mathbf{v}{⟩}_A$ is automatically symmetric (i.e. it satisfies the symmetry rule) because $A$ is symmetric:

$$\begin{array}{rl}⟨\mathbf{u},\mathbf{v}{⟩}_A& ={\mathbf{u}}^{\mathsf{T}}A\mathbf{v}\\ & =({\mathbf{u}}^{\mathsf{T}}A\mathbf{v}{)}^{\mathsf{T}}({\lambda }^{\mathsf{T}}=\lambda \text{ for scalars})\\ & ={\mathbf{v}}^{\mathsf{T}}{A}^{\mathsf{T}}({\mathbf{u}}^{\mathsf{T}}{)}^{\mathsf{T}}\\ & ={\mathbf{v}}^{\mathsf{T}}A\mathbf{u}\\ & =⟨\mathbf{v},A\mathbf{u}{⟩}_A.\end{array}$$

If you take any inner product given by a symmetric matrix, and plug the input $\mathbf{x}$ into both slots, you get a quadratic form:

$$⟨\mathbf{x},\mathbf{x}{⟩}_A={\mathbf{x}}^{\mathsf{T}}A\mathbf{x}=Q(\mathbf{x}).$$

The last property of inner products (namely that $⟨\mathbf{u},\mathbf{u}⟩>0$ etc.) means that a quadratic form created in this way by setting $Q(\mathbf{x})=⟨\mathbf{x},\mathbf{x}{⟩}_A$ is always a positive definite quadratic form.

Conversely, any inner product is given by some positive definite symmetric matrix. To find the matrix corresponding to an inner product, simply set $a_{ij}=⟨{\mathbf{e}}_i,{\mathbf{e}}_j⟩$. This works because $A{\mathbf{e}}_j$ gives the $j^{\text{th}}$ column vector of $A$, and then ${\mathbf{e}}_i^{\mathsf{T}}A{\mathbf{e}}_j$ gives the $i^{\text{th}}$ row entry of this column, which is therefore the $i^{\text{th}}$ row and $j^{\text{th}}$ column entry of $A$. By linearity, $⟨\cdot ,\cdot ⟩$ is then determined on every other pair of vectors by this data.

Addendum - determinants and eigenvectors

The determinant of a $2\times 2$ matrix is given by the formula:

$$A=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right),\det A=\left|\begin{array}{cc}a& b\\ c& d\end{array}\right|=ad-bc.$$

The determinant of a $3\times 3$ matrix is given by the formula:

$$A=\left(\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right),\det A=\left|\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right|=a\left|\begin{array}{cc}e& f\\ h& i\end{array}\right|-b\left|\begin{array}{cc}d& f\\ g& i\end{array}\right|+c\left|\begin{array}{cc}d& e\\ g& h\end{array}\right|.$$

Higher determinants

You may notice a pattern here that hold true in general and allows calculation of $n\times n$ determinants: traverse any row or column, using these entries as weights (with alternating signs), writing a weighted sum of the smaller determinants of the minor matrices (which are given by deleting the row and column in which the given entry in our traversal process lives). This process therefore reduces an $n\times n$ determinant to a linear combination of $(n-1)\times (n-1)$ determinants, and the reduction can be iterated until the $2\times 2$ case is reached.

The characteristic polynomial ${\mathrm{ch}}_A(\lambda )$ of a matrix $A$ is defined as the determinant $\det A_{\lambda }$ in which $\lambda$ is considered an indefinite variable.

For example: if $A=\left(\begin{array}{cc}1& -2\\ -4& -1\end{array}\right)$ then $A_{\lambda }=\left(\begin{array}{cc}1-\lambda & -2\\ -4& -1-\lambda \end{array}\right)$ and (using the determinant formula) we have:

$${\mathrm{ch}}_A(\lambda )=\left|\begin{array}{cc}1-\lambda & -2\\ -4& -1-\lambda \end{array}\right|=(1-\lambda )(-1-\lambda )-8={\lambda }^2-9.$$

Key fact for finding eigenvalues

The eigenvalues of a matrix $A$ are the roots of its characteristic polynomial ${\mathrm{ch}}_A(\lambda )$.

In the previous example, therefore, $\lambda =\pm 3$ are the eigenvalues.

Once the eigenvalues are found by solving ${\mathrm{ch}}_A(\lambda )=0$ for $\lambda$, the solutions can be plugged back in for $\lambda$ in the equation $A_{\lambda }\mathbf{x}=0$, and then we can try to find the eigenvectors $\mathbf{x}$ by solving this equation. For the $2\times 2$ case, you can solve it using the fact that one row is a multiple of the other. In $3\times 3$ and higher, you may need to perform row reduction on $A_{\lambda }$ (with $\lambda =\text{specific eigenvalue}$) in order to find solutions $\mathbf{x}$.

Notice that if $\mathbf{x}$ is an eigenvector, then $\alpha \mathbf{x}$ is also an eigenvector for any scalar $\alpha \ne 0$. So really we are studying eigenlines, namely the spans of eigenvectors. The relation of proportionality between $x_1$ and $x_2$ gives the direction of this line.

For $\lambda =+3$ in the previous example, solving $A_{\lambda }\mathbf{x}=0$ would mean solving $\left(\begin{array}{cc}-2& -2\\ -4& -4\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$. Therefore $-x_1-x_2=0$ and thus $x_1=-x_2$, so an eigenvector is given by $\left(\begin{array}{c}1\\ -1\end{array}\right)$ and a line of eigenvectors is given by the span $\alpha \left(\begin{array}{c}1\\ -1\end{array}\right)=\left(\begin{array}{c}\alpha \\ -\alpha \end{array}\right)$ for all $\alpha$. For $\lambda =-3$ we find $\left(\begin{array}{cc}4& -2\\ -4& 2\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$, and solving this we find $2x_1=x_2$ and therefore $\left(\begin{array}{c}1\\ 2\end{array}\right)$ is an eigenvector, as is everything in the span $\left(\begin{array}{c}\alpha \\ 2\alpha \end{array}\right)$.