# Tokenization

Canonical: https://aimastermindlab.com/glossary/tokenization

---
Tokenization is the process of splitting text into smaller units called tokens that an LLM can process. Tokens are typically word fragments, not whole words. Most English text averages about 0.75 words per token, so 1,000 tokens ≈ 750 words.

## What is Tokenization?

Tokenization converts raw text into a sequence of integer IDs that an LLM can process. Each integer maps to a **token** — a chunk of text that may be a full word, a subword fragment, a punctuation mark, or even a single character.

## Byte-Pair Encoding (BPE)

The dominant tokenization algorithm for modern LLMs. BPE starts with individual characters and iteratively merges the most frequent adjacent pairs until a target vocabulary size is reached (typically 32,000–100,000 tokens).

## Why Tokenization Matters for Developers

- **Cost** — APIs charge per token. Knowing token counts lets you estimate costs and optimise prompts.
- **Context limits** — Every model has a maximum context window measured in tokens. Exceeding it truncates your input.
- **Non-English languages** — Languages with less training data require more tokens per word, making them more expensive and less capable per model capacity unit.
- **Code** — Code tokenizes differently from prose, often requiring more tokens per meaningful unit.

## Counting Tokens

OpenAI uses `tiktoken` for GPT models. You can count tokens programmatically:

```python
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Hello, world!")
print(len(tokens))  # 4
```

## Useful Token Benchmarks

- 1 token ≈ 4 characters in English
- 1,000 tokens ≈ 750 words
- A page of text ≈ 500–800 tokens
- GPT-4o context: 128,000 tokens ≈ ~100,000 words

## Example

The word 'unbelievable' might be tokenized as ['un', 'believ', 'able'] — three tokens. 'AI' is one token. A 10,000-word document is roughly 13,000 tokens.
---

Source: https://aimastermindlab.com/glossary/tokenization — AI Mastermind Lab (https://aimastermindlab.com)