Words as points: text embeddings

Give words coordinates

An image can become a list of pixel values. We can also represent a token—a word or part of a word—by a list of numbers. Such a vector is called an embedding.

For a toy sentence with three tokens, stack their vectors as rows:

Read across to find one token's coordinates. Read down to compare one coordinate across all tokens. If there are $n$ tokens and $d$ coordinates per embedding, the matrix has shape $n\times d$.

Geometry can carry relationships

Once tokens have coordinates, we can compare them geometrically. Their dot product measures a combination of length and alignment. For nonzero vectors, dividing out their lengths gives cosine similarity:

This is useful when the representation has been learned so that the relationships we care about appear in its geometry. The coordinates are learned numbers, not necessarily axes we can name “animal,” “size,” or “emotion.” Contextual models can give the same token different vectors in different sentences.

A simpler picture is a projection

To draw high-dimensional embeddings on paper, keep only a few coordinates or project onto a few directions. This gives a view of the data, with some information left out. PCA chooses directions that capture the greatest variance after centering the data; those directions need not be the ones most relevant to a particular meaning or task.

Read the shape

A sentence has 12 tokens, each represented by 64 numbers. What is the matrix shape?

Answer: $12\times64$. It contains 12 points in a 64-dimensional space.