# vLLM Quick-Start: Serve Any LLM in 5 Minutes

> Learn how to deploy an open-source LLM for high-throughput production serving using vLLM and PagedAttention, complete with an OpenAI-compatible API.

Published: 2026-03-08  
Canonical: https://aimastermindlab.com/tutorials/vllm-quick-start-serve-llm

---
> **TL;DR:** If you want to deploy a model in production, don't use raw PyTorch or HuggingFace Transformers. Use **vLLM**. It utilizes "PagedAttention" to manage memory efficiently, resulting in up to 24x higher throughput. This guide shows you how to spin up a production-ready, OpenAI-compatible API server using vLLM in minutes.

When developing AI applications, tools like Ollama or LM Studio are incredible for local testing. But when it's time to deploy your application to the cloud (e.g., AWS, GCP, RunPod) to serve hundreds of concurrent users, you need an enterprise-grade inference engine.

[vLLM](https://github.com/vllm-project/vllm) is the industry standard for fast, high-throughput LLM serving. 

## Why vLLM?

The bottleneck in serving LLMs is rarely math (computation); it's memory bandwidth. Specifically, the "KV Cache" (the memory the model uses to remember context) becomes fragmented and eats up GPU VRAM instantly.

**PagedAttention**, the core innovation of vLLM, manages this memory exactly like an operating system manages virtual memory. It dramatically reduces bloat, allowing you to batch far more requests simultaneously without Out-Of-Memory (OOM) crashes.

## Prerequisites

- A Linux machine (Ubuntu recommended).
- An NVIDIA GPU (Compute Capability 7.0 or higher). Ampere (A10g, A100) or Hopper (H100) are ideal.
- NVIDIA Drivers and CUDA toolkit installed.
- Python 3.9+

## Step 1: Install vLLM

vLLM can be installed easily via pip. It's recommended to do this inside a fresh virtual environment or Conda environment.

```bash
pip install vllm
```

*(Note: Building vLLM from source can take a long time, so using the pre-compiled pip wheel is highly recommended unless you have specific hardware needs).*

## Step 2: Start the OpenAI-Compatible Server

The easiest way to use vLLM is via its built-in API server. It natively implements the exact same API format as OpenAI. This means any app you've built using the OpenAI SDK can be instantly pointed to your vLLM server by just changing the base URL.

Let's serve the highly capable `Llama-3-8B-Instruct` model from Hugging Face:

```bash
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --port 8000
```

### Understanding the Flags:
- `--model`: The Hugging Face model ID. (Note: For models like Llama 3, you must have accepted the license on Hugging Face and run `huggingface-cli login` first).
- `--tensor-parallel-size 1`: The number of GPUs to split the model across. If you have 2 GPUs and are serving a massive 70B model, set this to 2.
- `--max-model-len`: Caps the context window to save memory. 8192 is a safe default for 8B models on 24GB GPUs.
- `--port`: The port to run the HTTP server on.

When you run this command, vLLM will automatically download the weights, initialize the PagedAttention memory profiler, and start the server.

## Step 3: Query the Server

Once the server says "Uvicorn running on http://0.0.0.0:8000", you can ping it exactly as you would ping OpenAI.

### Using cURL:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is PagedAttention?"}
    ]
  }'
```

### Using the OpenAI Python SDK:

You don't need a special vLLM library. Just use the standard `openai` package!

```python
from openai import OpenAI

# Point the client to your vLLM server instead of standard OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY" # vLLM doesn't require a key by default
)

completion = client.chat.completions.create(
  model="meta-llama/Meta-Llama-3-8B-Instruct",
  messages=[
    {"role": "user", "content": "Write a python script to reverse a string."}
  ]
)

print(completion.choices[0].message.content)
```

## Step 4: Running vLLM in Docker (Recommended for Production)

Handling python dependencies on bare-metal servers can be messy. For true production environments, run vLLM via their official Docker container.

```bash
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model meta-llama/Meta-Llama-3-8B-Instruct
```

*The `-v ~/.cache/huggingface...` flag ensures that if your container restarts, it doesn't have to re-download the 15GB model weights!* 

## Performance Optimization Tips

If you see OOM (Out of Memory) errors or want to squeeze more speed out of your server:

1. **Use AWQ Quantization**: Appending `--quantization awq` to your startup command (and pointing to an AWQ version of a model, like `TheBloke/Llama-2-7B-Chat-AWQ`) will cut VRAM usage in half while maintaining speed.
2. **Tweak `--gpu-memory-utilization`**: By default, vLLM claims 90% of your GPU's RAM right on startup to reserve a massive KV cache pool. If you are running multiple things on the GPU, you can lower this to e.g., `0.5` (50%).
3. **Handle Too Many Requests**: If users send a burst of traffic, vLLM inherently handles the queueing. Requests might take slightly longer, but the server won't crash.
---

Source: https://aimastermindlab.com/tutorials/vllm-quick-start-serve-llm — AI Mastermind Lab (https://aimastermindlab.com)