Tokenisation

The idea#

A model computes with numbers. Before text goes in, it is cut into pieces and each piece gets a number. Those pieces are called tokens and are usually shorter than a word and longer than a letter.

The German "Rechnungsprüfung" becomes roughly Rech | nungs | prüfung, three tokens. The English "invoice check" is two.

What follows#

Byte-pair encoding#

1. 01

   Start with individual bytes

   The vocabulary initially contains all 256 possible bytes. Every text is representable, including emoji and other scripts.

2. 02

   Find the most frequent pair

   Across the whole training text, find the most frequently adjacent pair.

3. 03

   Merge

   That pair becomes a new token with its own number.

4. 04

   Repeat

   Until the target vocabulary size is reached, usually 32,000 to 200,000.

# Rules of thumb for a quick estimate, without a library:
def estimate_tokens(text, language="en"):
    chars_per_token = 3.2 if language == "de" else 4.0
    return round(len(text) / chars_per_token)

de = "Die Rechnungsprüfung erfolgt durch die Fachabteilung."
en = "The invoice check is carried out by the department."
print(estimate_tokens(de, "de"), estimate_tokens(en, "en"))   # 16  13

What this means for cost#

At 3 EUR per million input tokens and a document of 10,000 characters:

The 25 to 43 percent premium is not a rounding error but a structural disadvantage that belongs in every cost estimate.

Why perplexity becomes incomparable#

Perplexity is measured per token. A tokeniser that splits the same text into more pieces spreads the same information over more predictions and lowers perplexity without the model being better.

Worked through: model A has PPL = 8 at 3.2 characters per token; model B has PPL = 12 at 4.0.

• A: (1/3.2) × log₂ 8 = 0.3125 × 3 = 0.938 bits per character
• B: (1/4.0) × log₂ 12 = 0.25 × 3.585 = 0.896 bits per character

Model B is the better language model despite the higher perplexity. A raw perplexity comparison would have suggested the opposite.

Practical traps#

Vocabulary size as a trade-off#

At h = 4096 the output layer costs 131 million parameters at 32,000 entries and 1.05 billion at 256,000. A larger vocabulary saves context and compute per text but costs memory: the trade every model family decides afresh.