Vector Subspaces

keep in mind that our discussions would only involve the field

If is a vector space over then a subset of is said to be a subspace of vector space , if is also a vector space over the same field . (The definition holds for any vector space V over field F but in machine learning the vector space is more common.)

An alternative definition, these three imply all six of a vector space’s properties as discussed in a previous note:
If is a vector space, then is said to be a subspace of if:

• is closed under vector addition, so implies
• is closed under scalar multiplication, so implies

Theorem that follows the alternative definition:
Let W be a subset of . Then W is a subspace of if and only if the following conditions hold:
a) W is non empty (we are looking for zero vector here)
b) For any and any :

examples:

• Let be a vector space, then and are subspaces of and are called trivial subspaces.
• The set is a subspace of
  • Let’s try to show it. The zero vector of is given as . This is a vector in as . Therefore, is non-empty.
  • Let and be vectors in . Thus and . We have to show that . . Now is ? We can do on and check if it’s 0. Thus because recall that so
• similarly, the set is a subspace of
• is NOT a subspace of
  • Let’s try to show it. First, the zero vector for is . So even the first condition is not satisfied as
• If we have a subspace of , then geometrically:
  • it’s the origin
  • it’s a line passing through the origin
  • it’s a plane passing through the origin
  • or itself. The first and last conditions are “trivial”.

Some results:

1. The intersection of a non-empty collection of subspaces of is a subspace of ; or if are subspaces of , is a subspace of V.
2. The union of two subspaces of is a subspace of iff one of them is contained in the other

Linear independence

keep in mind that sometimes I have represented vectors without bold or an arrow, but generally any “element” in S is a vector as it’s a set of vectors.

Consider a vector space and a finite number of vectors . Then every of the form:

with is a linear combination of vectors .

Let us consider a vector space with and . If there is a non-trivial linear combination such that with at least one , the vectors are linearly dependent. If only the trivial solution exists, i.e. the vectors are linearly independent.
Some properties to find whether vectors are linearly independent:

• k vectors are either linearly dependent or independent, there is no third option
• if at least one of the vectors in is , then they are linearly dependent. The same holds if two vectors are identical.
• A practical way of checking whether vectors are linearly independent is to use Gaussian elimination. Write all vectors as columns of matrix and perform Gaussian elimination until the matrix is in row-echelon form.
  • The pivot columns indicate the vectors which are linearly independent of the vectors on the left.
  • The non pivot columns can be expressed as linear combinations of the pivot columns on their left.
  • ex. in row echelon form we can say that the first and third columns are pivot columns. The second column is a non pivot column as it’s three times the first column.
  • All column vectors are linearly independent iff all columns are pivot columns. If there is at least one non pivot column, the columns and thus the corresponding vectors are linearly dependent.
    Keep in mind that we use Gaussian elimination because we follow the general approach and solve .
    ex. consider with

to check whether they are linearly independent, we solve:

for the row echelon form step

here every column is a pivot column and so there is no non-trivial solution and we require to solve the equation system. Hence vectors are linearly independent.

span or linear span

Let be a vector space over field F. Let be a non-empty subset of . Then the set is a linear span of set . The span of S is also denoted by and is the set of all finite linear combinations of elements (vectors) of .

ex. consider the set . which is nothing but . Thus the linear span of the given set was .

ex. consider the set ; which is which is also a subspace of .

some results:

1. Let V(F) be a vector space and let S be a non-empty subset of V. Then L(S) is a subspace of V.

   • proof: the span of S is . Now, the additive identity 0 is in span(S) because , so the first condition is satisfied. Keep in mind that in this set the only element is . Let’s now take two elements of this set and add them to check closure: . Thus addition is closed. Also, let’s take a scalar and multiply it with an element: . Thus scalar multiplication is also closed and hence the span is a subspace of V.

2. Let S be a non-empty subset of a vector space V. Then L(S) is the smallest subspace of V(F) containing S.

   • proof: we know that the span(S) is a subspace of V by the previous proof. Now, consider the fact that contains each vector in because if you take an arbitrary vector it’s a linear combination of other vectors (set all other ‘s to 0 and to 1). Also consider that because subspaces are closed under scalar multiplication and addition, every subspace of V that contains each also has to contain because it too contains each . Thus span is the smallest subspace of V containing all the vectors or S.

Some important subspaces for dimensional reduction

let the matrix

where . Then:

1. The row space of given by is a subspace of
2. The column space of given by is a subspace of
3. The set is said to be the nullspace of .
4. The set is said to be the range of A.
   In general, column space of A = range of A.