Transformer Tokens & Embeddings

Transformers read and write sequences of tokens. But what is a token sequence?

How can transformer models input and output text, images, video, MIDI and audio?

Tokenizer Example

To train or use a transformer, we need to

Here's an extremely simple text tokenizer:

This simple tokenizer has limited value. It can only encode and decode the 68 characters in its vocabulary:

 abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ.!?"'0123456789

What is a token?

Assuming a vocabulary size of xx, a token is just an integer value that satisfies:

0≤token<x0 \leq token < x

Remember NotaGen from the Symbolic Music Generation assignment? NotaGen generates abc notation, which is just text. Now we know one way that models can produce text.

Byte Pair Encoding

Text-based LLMs typically use Byte Pair Encoding (BPE), which tokenizes groups of characters.

A robust BPE text tokenizer will help the model see common subword character sequences. In English "izer" is a common subword, so a BPE tokenizer might split "tokenizer" into tokens like token, and izer (instead of e.g. toke and nizer). This helps models generalize and "understand" grammar.

Interactive Tokenizers

Music Tokenization

MIDI Tokenization

Audio Tokenization

Embeddings

Once we have tokens, the transformer will use them to look up column vectors from an embedding table. Each token index corresponds to a column vector that represents that token numerically.

(space)
↓
0
↓
[−1.0−0.9−1.0−1.4−0.9−0.6−0.0⋮−1.1]{\def\arraystretch{1}\begin{bmatrix} -1.0 \\ \phantom{-}0.9 \\ -1.0 \\ \phantom{-}1.4 \\ \phantom{-}0.9 \\ -0.6 \\ \phantom{-}0.0 \\ \vdots \\ -1.1 \end{bmatrix}}
a
↓
1
↓
[−1.0−1.3−1.7−1.6−0.5−1.6−1.7⋮−1.5]{\def\arraystretch{1}\begin{bmatrix} -1.0 \\ \phantom{-}1.3 \\ \phantom{-}1.7 \\ \phantom{-}1.6 \\ -0.5 \\ -1.6 \\ \phantom{-}1.7 \\ \vdots \\ -1.5 \end{bmatrix}}
b
↓
2
↓
[−1.0−1.7−0.4−1.7−1.8−1.4−0.6⋮−1.8]{\def\arraystretch{1}\begin{bmatrix} -1.0 \\ \phantom{-}1.7 \\ \phantom{-}0.4 \\ \phantom{-}1.7 \\ -1.8 \\ \phantom{-}1.4 \\ -0.6 \\ \vdots \\ -1.8 \end{bmatrix}}
c
↓
3
↓
[−1.0−2.0−0.8−1.9−0.9−0.4−1.2⋮−1.9]{\def\arraystretch{1}\begin{bmatrix} -1.0 \\ -2.0 \\ -0.8 \\ \phantom{-}1.9 \\ \phantom{-}0.9 \\ \phantom{-}0.4 \\ \phantom{-}1.2 \\ \vdots \\ \phantom{-}1.9 \end{bmatrix}}
 
 
 
 
⋯
9
↓
67
↓
[−0.9−1.2−1.5−0.7−0.5−0.2−0.5⋮−0.4]{\def\arraystretch{1}\begin{bmatrix} -0.9 \\ \phantom{-}1.2 \\ -1.5 \\ -0.7 \\ -0.5 \\ -0.2 \\ -0.5 \\ \vdots \\ \phantom{-}0.4 \end{bmatrix}}

During the training process, the values in the embedding table are gradually adjusted. If we train successfully, these values will encode the semantic significance of the tokens in a high-dimensional space.

Resources