Inference

Inference is the process of running a trained model on new data to generate predictions or outputs. In LLMs, inference refers to generating a response given a prompt. It is the production phase of the AI lifecycle — distinct from training, which updates model weights.

What is Inference?

In machine learning, inference is the process of using a trained model to generate predictions on new, unseen data. In the context of LLMs, every time you send a prompt and receive a response, you are performing inference.

Inference vs Training

Training Inference
Purpose Update model weights Generate predictions
Compute Massive (weeks/months on GPU clusters) Moderate (milliseconds to seconds)
Memory Requires gradients + activations Weights only (+ KV cache)
Frequency Once (or periodically) Billions of times per day in production

Key Inference Metrics

  • Latency — Time to first token (TTFT) and time to complete response.
  • Throughput — Tokens per second, requests per second.
  • VRAM — How much GPU memory the model requires at inference time.
  • Cost — $/1M tokens (cloud) or $/GPU-hour (self-hosted).

Optimising Inference

  • Quantization — Reduce weight precision to shrink memory and increase speed.
  • KV cache — Cache key-value attention states to avoid recomputation.
  • Batching — Process multiple requests together to maximise GPU utilisation.
  • Speculative decoding — Use a small draft model to propose tokens; large model verifies in parallel.
  • Flash Attention — Memory-efficient attention implementation.

Inference Providers

Managed inference: OpenAI API, Anthropic API, Google Vertex AI, AWS Bedrock. Open-source inference: Ollama (local), vLLM (production self-hosted), llama.cpp.

Example

Asking ChatGPT a question, running a Llama 3 model with Ollama on your laptop, or calling the OpenAI API in your application — all are inference. Training GPT-4 on OpenAI's supercomputer cluster is not inference.

Related terms

  • Quantization Quantization reduces the numerical precision of model weights (e.g. from 32-bit floats to 4-bit integers), shrinking memory requirements and accelerating inference with minimal quality loss. It's the key technique for running large models on consumer hardware.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • LoRA (Low-Rank Adaptation) LoRA is a parameter-efficient fine-tuning technique that trains only a small set of low-rank adapter matrices instead of updating all model weights. It reduces the number of trainable parameters by up to 10,000x while achieving fine-tuning quality comparable to full parameter updates.
  • Fine-tuning Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset to specialise its behaviour. It updates the model's weights, making it more accurate for a specific domain, style, or task without training from scratch.