Theory

Independence

A collection of vectors $\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_k\}$ is called independent when the only solution to the equation

$$x_1{\mathbf{v}}_1+x_2{\mathbf{v}}_2+\cdots +x_k{\mathbf{v}}_k=0$$

is given by setting $x_i=0$ for every $i$.

Independence (part 2)

Independence and dimension
Recall that the dimension of a subspace is equal to the smallest number of vectors needed to span the subspace.

If a set of vectors has the smallest number that spans whatever they span, then this set is independent. (If there is any dependency relation, then a smaller set will have the same span: just eliminate the dependent vector.)

Conversely, if some vectors are independent, then they are automatically the smallest number that spans their span:

Theorem: Independent vectors span their span minimally

Let $V=⟨{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k⟩\subset {\mathbb{R}}^n$ be the subspace spanned by the set of vectors $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$, and suppose the set $\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$ is independent. Then any set of vectors spanning $V$ must have at least $k$ elements.

Proof

We use Steinitz Exchange but with ${\mathbf{v}}_i$ in place of ${\mathbf{e}}_i$. Suppose $V=⟨{\mathbf{u}}_1,\dots ,{\mathbf{u}}_r⟩$ for $r$ other vectors ${\mathbf{u}}_j$. Our goal is to show that $r\ge k$.

First, since ${\mathbf{v}}_1\in V$, we know there is a linear combination

$${\mathbf{v}}_1=a_1{\mathbf{u}}_1+\cdots +a_r{\mathbf{u}}_r$$

with at least one coefficient $a_{i^{\star }}\ne 0$. Solve for ${\mathbf{u}}_{i^{\star }}$ in terms of the others in this equation to obtain the fact that $V=⟨{\mathbf{v}}_1,{\mathbf{u}}_2^{\prime },\dots ,{\mathbf{u}}_r^{\prime }⟩$, where we introduce the labels ${\mathbf{u}}_j^{\prime }$ with $j=2,\dots ,r$ for whichever vectors are left from $\left\{{\mathbf{u}}_1,\dots ,{\mathbf{u}}_r\right\}$ after removing ${\mathbf{u}}_{i^{\star }}$.

By iterating this process, we obtain

$$⟨{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_k,{\mathbf{u}}_{k+1}^{(k-1)},\dots ,{\mathbf{u}}_{r-k}^{(k-1)}⟩,$$

and this is only possible if $r\ge k$.

(Notation: here ${\mathbf{u}}_j^{(p)}$ stands for ${\mathbf{u}}_j^{\prime \prime }{}^{\dots }{}^{\prime }$ with $p$ ticks, just as for higher derivatives.)

We are able to iterate the process because the vectors ${\mathbf{v}}_i$ are independent: at each stage, the next ${\mathbf{v}}_{i+1}$ can be written as a linear combination of the previous ${\mathbf{v}}_j$ ($j<i$) and some other ${\mathbf{u}}_j$. But the coefficient on at least one of the ${\mathbf{u}}_j$ must be nonzero, otherwise we have written ${\mathbf{v}}_i$ in terms of ${\mathbf{v}}_j$ alone, and that is impossible by independence.

Basis

A basis for $V\subset {\mathbb{R}}^n$ is a set of independent vectors which span the entire subspace $V$.

For example, the standard basis $\{{\mathbf{e}}_1,\dots ,{\mathbf{e}}_n\}$ is a basis for the whole space ${\mathbb{R}}^n$.

The criterion of independence is a function of the relationships between the vectors in the set. The criterion of spanning also depends upon what other vectors there may be in the subspace.

(Independence is an intrinsic concept, whereas spanning the subspace is an extrinsic concept. These are not precise mathematical terms.)

Any set of vectors spans a certain subspace, namely the span itself. Therefore any independent set of vectors is a basis for its own span.

Because an independent set of vectors is always the minimal number needed to span its span (the theorem above), and a basis for a given space is an independent set of vectors spanning the subspace, every basis for a given subspace has the same number of vectors. (Since every basis has the minimal number needed to span the given subspace.) This number as the dimension of the subspace.

The most important way to think about and use the concept of basis is this: Given a basis $\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_k\}$ for a subspace $V\subset {\mathbb{R}}^n$, every vector $\mathbf{w}$ in $V$ can be obtained uniquely as a linear combination of basis vectors:

$$\mathbf{w}=x_1{\mathbf{v}}_1+\cdots +x_k{\mathbf{v}}_k.$$

(Here ‘unique’ means that the coefficients $x_i$ are uniquely determined by $\mathbf{w}$.)

From this way of thinking about a basis, we see that the dimension $k$ is the number of quantities $x_i$ that are needed to describe everything in the space without redundancy.

Basis (part 2)

Orthonormal systems, orthonormal bases
A collection of vectors $\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots ,{\mathbf{v}}_k\}$ is called an orthonormal system when they are:

• unit vectors, meaning $|{\mathbf{v}}_i|=1$ for all $i$
• pairwise orthogonal, meaning ${\mathbf{v}}_i\cdot {\mathbf{v}}_j=0$ when $i\ne j$

If the collection is also a basis, then it is called an orthonormal basis.

Vector components in a given basis
The typical application of a basis is to write other vectors as its linear combinations. Since a basis spans the whole space, every other vector in the space can be written using some set of coefficients. Since a basis is linearly independent, there is a unique set of coefficients that will generate a given vector.

For example, if $\mathcal{B}=\{{\mathbf{b}}_1,\dots ,{\mathbf{b}}_k\}$ is a basis of a subspace $V\subset {\mathbb{R}}^n$, which incidentally therefore has dimension $k$, then we can write any $\mathbf{v}\in V$ by:

$$\mathbf{v}=x_1{\mathbf{b}}_1+x_2{\mathbf{b}}_2+\cdots +x_k{\mathbf{b}}_k.$$

Using such linear combinations to express vectors $\mathbf{v}$, we find a unique association between a vector $\mathbf{v}$ and the quantities $x_1,\dots ,x_k$ which express that vector in terms of the ${\mathbf{b}}_i$. We can group those quantities into a list $(x_1,\dots ,x_k)$. The terms are called the components of $\mathbf{v}$ in the basis $\mathcal{B}$.

The ordinary component entries of a vector are actually just the components of the vector in the standard basis. (Think about this!)