LoRA (Low-Rank Adaptation)

LoRA is a parameter-efficient fine-tuning technique that trains only a small set of low-rank adapter matrices instead of updating all model weights. It reduces the number of trainable parameters by up to 10,000x while achieving fine-tuning quality comparable to full parameter updates.

What is LoRA?

Low-Rank Adaptation (LoRA) is a fine-tuning method introduced by Hu et al. (2021) that dramatically reduces the computational cost of adapting large language models. Instead of updating all of a model's billions of parameters, LoRA freezes the original weights and adds small, trainable low-rank decomposition matrices to each layer.

The Core Idea

For a weight matrix W (shape d×k), LoRA adds ΔW = B×A where:

  • B has shape d×r
  • A has shape r×k
  • r is the rank (typically 4–64, far smaller than d or k)

Training only A and B reduces trainable parameters massively — a 7B parameter model might require fine-tuning only 4–20M parameters with LoRA.

QLoRA

QLoRA (Quantized LoRA) applies LoRA to a 4-bit quantized base model, further reducing memory requirements. QLoRA makes fine-tuning 70B parameter models feasible on a single consumer GPU (e.g. 24GB VRAM).

LoRA Hyperparameters

  • rank (r) — Controls the expressivity of the adapters. Higher ranks can learn more but risk overfitting.
  • alpha (α) — Scaling factor for the adapter output. Usually set equal to rank.
  • target modules — Which layers to adapt. Query and value projection matrices are common choices.

LoRA Variants

  • DoRA — Decomposes weights into magnitude and direction for better fidelity.
  • AdaLoRA — Adaptively allocates rank budget across layers.
  • LoftQ — Initialises LoRA adapters in a way that better compensates for quantization error.

Example

A healthcare company wants GPT-style responses tuned to medical terminology. Using QLoRA, they fine-tune a 13B parameter Llama 3 model on 500 clinical note examples on a single A100 GPU in under 2 hours — something that would have required a cluster with full fine-tuning.

Related terms

  • Fine-tuning Fine-tuning is the process of continuing to train a pre-trained model on a smaller, task-specific dataset to specialise its behaviour. It updates the model's weights, making it more accurate for a specific domain, style, or task without training from scratch.
  • Quantization Quantization reduces the numerical precision of model weights (e.g. from 32-bit floats to 4-bit integers), shrinking memory requirements and accelerating inference with minimal quality loss. It's the key technique for running large models on consumer hardware.
  • LLM (Large Language Model) A Large Language Model is a neural network trained on massive text corpora to predict and generate text. LLMs like GPT-4, Claude, and Gemini exhibit broad language understanding, reasoning, and generation capabilities that emerge from scale.
  • RLHF (Reinforcement Learning from Human Feedback) RLHF is the training technique used to align LLMs with human preferences. Human annotators rank model outputs, those preferences train a reward model, and the LLM is then fine-tuned using reinforcement learning to maximise the reward signal. RLHF is how ChatGPT was trained to be helpful and safe.