# Quantization

Canonical: https://aimastermindlab.com/glossary/quantization

---
Quantization reduces the numerical precision of model weights (e.g. from 32-bit floats to 4-bit integers), shrinking memory requirements and accelerating inference with minimal quality loss. It's the key technique for running large models on consumer hardware.

## What is Model Quantization?

Quantization reduces the bit-width used to represent model weights. Instead of storing each parameter as a 32-bit float (FP32) or 16-bit float (FP16/BF16), quantized models store weights as 8-bit integers (INT8) or even 4-bit integers (INT4).

## Why Quantize?

- **Memory** — A 7B parameter model in FP16 requires ~14GB VRAM. In 4-bit, it requires ~4GB — runnable on a laptop GPU.
- **Speed** — Integer arithmetic is faster than floating-point on many hardware backends.
- **Cost** — Smaller models are cheaper to serve at scale.

## Common Quantization Methods

| Method | Bit-width | Quality | Tool |
|---|---|---|---|
| GGUF (Q4_K_M) | 4-bit | Good | llama.cpp |
| GPTQ | 4-bit | Very good | AutoGPTQ |
| AWQ | 4-bit | Excellent | AutoAWQ |
| GGUF (Q8_0) | 8-bit | Near-lossless | llama.cpp |
| bitsandbytes | 4/8-bit | Good | Hugging Face |

## Post-Training vs Quantization-Aware Training

- **Post-training quantization (PTQ)** — Quantize after training. Fast, but some quality loss.
- **Quantization-aware training (QAT)** — Train with simulated quantization. Better quality, more expensive.

## Running Quantized Models Locally

The most common workflow uses **llama.cpp** with GGUF format models downloaded from Hugging Face. Tools like **Ollama** and **LM Studio** wrap llama.cpp with a user-friendly interface.

## Example

Meta's LLaMA 3 70B at FP16 requires ~140GB VRAM — 2× A100 GPUs. At Q4_K_M quantization it fits in ~45GB — feasible on a single A100 or two cheaper GPUs — with only marginal quality degradation on most benchmarks.
---

Source: https://aimastermindlab.com/glossary/quantization — AI Mastermind Lab (https://aimastermindlab.com)