Quantization

Quantization reduces the numerical precision of model weights (e.g. from 32-bit floats to 4-bit integers), shrinking memory requirements and accelerating inference with minimal quality loss. It's the key technique for running large models on consumer hardware.

What is Model Quantization?

Quantization reduces the bit-width used to represent model weights. Instead of storing each parameter as a 32-bit float (FP32) or 16-bit float (FP16/BF16), quantized models store weights as 8-bit integers (INT8) or even 4-bit integers (INT4).

Why Quantize?

  • Memory — A 7B parameter model in FP16 requires ~14GB VRAM. In 4-bit, it requires ~4GB — runnable on a laptop GPU.
  • Speed — Integer arithmetic is faster than floating-point on many hardware backends.
  • Cost — Smaller models are cheaper to serve at scale.

Common Quantization Methods

Method Bit-width Quality Tool
GGUF (Q4_K_M) 4-bit Good llama.cpp
GPTQ 4-bit Very good AutoGPTQ
AWQ 4-bit Excellent AutoAWQ
GGUF (Q8_0) 8-bit Near-lossless llama.cpp
bitsandbytes 4/8-bit Good Hugging Face

Post-Training vs Quantization-Aware Training

  • Post-training quantization (PTQ) — Quantize after training. Fast, but some quality loss.
  • Quantization-aware training (QAT) — Train with simulated quantization. Better quality, more expensive.

Running Quantized Models Locally

The most common workflow uses llama.cpp with GGUF format models downloaded from Hugging Face. Tools like Ollama and LM Studio wrap llama.cpp with a user-friendly interface.

Example

Meta's LLaMA 3 70B at FP16 requires ~140GB VRAM — 2× A100 GPUs. At Q4_K_M quantization it fits in ~45GB — feasible on a single A100 or two cheaper GPUs — with only marginal quality degradation on most benchmarks.

Related terms

  • LoRA (Low-Rank Adaptation) LoRA is a parameter-efficient fine-tuning technique that trains only a small set of low-rank adapter matrices instead of updating all model weights. It reduces the number of trainable parameters by up to 10,000x while achieving fine-tuning quality comparable to full parameter updates.
  • Fine-tuning Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset to specialise its behaviour. It updates the model's weights, making it more accurate for a specific domain, style, or task without training from scratch.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • Inference Inference is the process of running a trained model on new data to generate predictions or outputs. In LLMs, inference refers to generating a response given a prompt. It is the production phase of the AI lifecycle — distinct from training, which updates model weights.