Alex Loftus
← Back to the lesson

Rank, linear combinations, and information loss

The vectors u=(1,2) and w=(2,−1) provide independent directions. Here 1.0u−0.6w reaches (−0.2,2.6), outside the line through u. Changing the two coefficients can reach any point in the plane.
The vectors u=(1,2) and w=(2,−1) provide independent directions. Here 1.0u−0.6w reaches (−0.2,2.6), outside the line through u. Changing the two coefficients can reach any point in the plane.

How many directions can the output reach?

A matrix turns input coefficients into combinations of its columns. The rank counts how many independent output directions those columns provide.

If every column is a multiple of one arrow and at least one column is nonzero, the outputs fill one line: rank one. Two non-collinear columns can reach a plane: rank two. Rank is about the span of the columns, not simply the number of columns written down.

One vector can build a rank-one matrix

Take g=(1,2)g=(1,2) as a column. Its outer product is

ggT=[1224].gg^\mathsf{T}=\begin{bmatrix}1&2\\2&4\end{bmatrix}.

The second column is twice the first. No combination of those columns escapes their line. For any nonzero real gg, ggTgg^\mathsf{T} has rank one; for g=0g=0, its rank is zero.

Projection shows how information disappears

Consider the map that drops height:

P(x,y,z)=(x,y,0).P(x,y,z)=(x,y,0).

The points (2,1,3)(2,1,3) and (2,1,−4)(2,1,-4) land on the same point. Once height is discarded, the output alone cannot tell us which input we had.

The null space contains all inputs that become zero. Here it is the entire vertical axis:

null⁡(P)={(0,0,z):z∈R}.\operatorname{null}(P)=\{(0,0,z):z\in\mathbb R\}.

For an m×nm\times n matrix, rank plus null-space dimension equals nn. In this example, 2+1=32+1=3: two input directions survive and one disappears.

Can a full-rank matrix lose information?

Answer: Yes, if it is wide. A 2×32\times3 matrix can have the maximum possible rank 2 while still having a one-dimensional null space. A square full-rank matrix is the case where an inverse recovers every input.