# Inference

Canonical: https://aimastermindlab.com/glossary/inference

---
Inference is the process of running a trained model on new data to generate predictions or outputs. In LLMs, inference refers to generating a response given a prompt. It is the production phase of the AI lifecycle — distinct from training, which updates model weights.

## What is Inference?

In machine learning, inference is the process of using a **trained model** to generate predictions on new, unseen data. In the context of LLMs, every time you send a prompt and receive a response, you are performing inference.

## Inference vs Training

| | Training | Inference |
|---|---|---|
| Purpose | Update model weights | Generate predictions |
| Compute | Massive (weeks/months on GPU clusters) | Moderate (milliseconds to seconds) |
| Memory | Requires gradients + activations | Weights only (+ KV cache) |
| Frequency | Once (or periodically) | Billions of times per day in production |

## Key Inference Metrics

- **Latency** — Time to first token (TTFT) and time to complete response.
- **Throughput** — Tokens per second, requests per second.
- **VRAM** — How much GPU memory the model requires at inference time.
- **Cost** — $/1M tokens (cloud) or $/GPU-hour (self-hosted).

## Optimising Inference

- **Quantization** — Reduce weight precision to shrink memory and increase speed.
- **KV cache** — Cache key-value attention states to avoid recomputation.
- **Batching** — Process multiple requests together to maximise GPU utilisation.
- **Speculative decoding** — Use a small draft model to propose tokens; large model verifies in parallel.
- **Flash Attention** — Memory-efficient attention implementation.

## Inference Providers

Managed inference: OpenAI API, Anthropic API, Google Vertex AI, AWS Bedrock.
Open-source inference: Ollama (local), vLLM (production self-hosted), llama.cpp.

## Example

Asking ChatGPT a question, running a Llama 3 model with Ollama on your laptop, or calling the OpenAI API in your application — all are inference. Training GPT-4 on OpenAI's supercomputer cluster is not inference.
---

Source: https://aimastermindlab.com/glossary/inference — AI Mastermind Lab (https://aimastermindlab.com)