LLM (Large Language Model)

A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.

What is a Large Language Model?

A Large Language Model (LLM) is a neural network — typically a Transformer — trained on hundreds of billions to trillions of words of text to learn the statistical patterns of language. At sufficient scale, LLMs develop emergent capabilities like reasoning, instruction following, code generation, and multilingual understanding.

Scale and Emergence

The defining characteristic of LLMs is scale — both in parameters (billions to trillions) and training data. Below certain sizes, models exhibit limited capabilities. Above certain thresholds, new capabilities emerge that were not present in smaller models. This non-linear relationship between scale and capability was formalized in the scaling laws research from OpenAI and DeepMind.

How LLMs Generate Text

LLMs are next-token predictors. Given a sequence of tokens, the model assigns a probability distribution over its entire vocabulary to predict the most likely next token. Repeating this process autoregressively generates fluent, coherent text.

Modern LLM Families

Family Creator Notable Models
GPT OpenAI GPT-4o, o1, o3
Claude Anthropic Claude 3.5 Sonnet, Claude 3 Opus
Gemini Google DeepMind Gemini 1.5 Pro, Gemini Flash
LLaMA Meta AI LLaMA 3 8B, 70B, 405B
Mistral Mistral AI Mistral Large, Mixtral 8x22B
Qwen Alibaba Qwen2.5-72B

Closed vs Open Source

Closed LLMs (GPT-4, Claude) are accessible only via API. Open-weight models (LLaMA, Mistral) can be downloaded and run locally, offering privacy, cost savings, and full customisation at the cost of infrastructure management.

Example

When you ask ChatGPT to write a Python script, debug your code, or summarise a document, you're interacting with a large language model that generates each word token-by-token based on what is statistically most likely given your prompt.

Related terms

  • Transformer The Transformer is a neural network architecture introduced by Google in 2017 that uses self-attention to process sequences in parallel. It is the foundation of virtually all modern large language models, including GPT, Claude, Gemini, and LLaMA.
  • Fine-tuning Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset to specialise its behaviour. It updates the model's weights, making it more accurate for a specific domain, style, or task without training from scratch.
  • RAG (Retrieval-Augmented Generation) RAG is a technique that combines an LLM with an external knowledge retrieval system. Instead of relying solely on its training data, the model retrieves relevant documents at query time and uses them as context to generate more accurate, up-to-date responses.
  • Context Window The context window is the maximum amount of text (measured in tokens) that an LLM can process in a single inference call. Everything outside the window is effectively invisible to the model. Modern models range from 8K to 2M+ token context windows.
  • Tokenization Tokenization is the process of splitting text into smaller units called tokens that an LLM can process. Tokens are typically word fragments, not whole words. Most English text averages about 0.75 words per token, so 1,000 tokens ≈ 750 words.