Tokenization

Tokenization is the process of splitting text into smaller units called tokens that an LLM can process. Tokens are typically word fragments, not whole words. Most English text averages about 0.75 words per token, so 1,000 tokens ≈ 750 words.

What is Tokenization?

Tokenization converts raw text into a sequence of integer IDs that an LLM can process. Each integer maps to a token — a chunk of text that may be a full word, a subword fragment, a punctuation mark, or even a single character.

Byte-Pair Encoding (BPE)

The dominant tokenization algorithm for modern LLMs. BPE starts with individual characters and iteratively merges the most frequent adjacent pairs until a target vocabulary size is reached (typically 32,000–100,000 tokens).

Why Tokenization Matters for Developers

  • Cost — APIs charge per token. Knowing token counts lets you estimate costs and optimise prompts.
  • Context limits — Every model has a maximum context window measured in tokens. Exceeding it truncates your input.
  • Non-English languages — Languages with less training data require more tokens per word, making them more expensive and less capable per model capacity unit.
  • Code — Code tokenizes differently from prose, often requiring more tokens per meaningful unit.

Counting Tokens

OpenAI uses tiktoken for GPT models. You can count tokens programmatically:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Hello, world!")
print(len(tokens))  # 4

Useful Token Benchmarks

  • 1 token ≈ 4 characters in English
  • 1,000 tokens ≈ 750 words
  • A page of text ≈ 500–800 tokens
  • GPT-4o context: 128,000 tokens ≈ ~100,000 words

Example

The word 'unbelievable' might be tokenized as ['un', 'believ', 'able'] — three tokens. 'AI' is one token. A 10,000-word document is roughly 13,000 tokens.

Related terms

  • Context Window The context window is the maximum amount of text (measured in tokens) that an LLM can process in a single inference call. Everything outside the window is effectively invisible to the model. Modern models range from 8K to 2M+ token context windows.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • Temperature Temperature is a hyperparameter that controls the randomness of an LLM's output. A temperature of 0 produces deterministic, predictable responses. Higher values (0.7–1.0+) produce more creative, varied, and sometimes unpredictable text.
  • Embedding An embedding is a numerical vector representation of data (text, image, audio) that captures semantic meaning. Similar concepts have embeddings that are close together in vector space, enabling AI systems to measure similarity, cluster data, and retrieve relevant information.