# Fine‑Tuning LLMs in 2026: LoRA, QLoRA, and vLLM

> When to fine‑tune vs RAG, adapter sizing defaults, evaluation you can trust, and fast serving with paged attention.

Published: 2026-03-01  
Canonical: https://aimastermindlab.com/knowledge-base/finetuning-llms-2026

---
Fine‑tuning gives an LLM new habits and stronger consistency for your specific product tasks; retrieval (RAG) gives it fresh facts and citations. In 2026, most reliable systems combine both.

## TL;DR
- Fine‑tune when you need new behavior, domain style, or strict output formats that prompting + RAG cannot reliably produce. Keep RAG for fresh or fast‑changing knowledge and for citations.
- Start with small, well‑curated instruction data. Use parameter‑efficient adapters (LoRA/QLoRA). Measure formatting accuracy and task quality, not just loss.
- Serve with an inference engine that supports paged/segmented KV‑cache and continuous batching (e.g., vLLM). Add speculative decoding to cut tail latency.

## When To Fine‑Tune vs. Use RAG
Use this quick decision guide:
- Need up‑to‑date facts, references, or long‑tail entities → Prefer RAG. Add strong retrieval + re‑ranking and cite sources.
- Need consistent structure (JSON, SQL, DSL), brand tone, or task‑specific steps → Fine‑tune a small adapter on examples showing exactly the target behavior.
- Hybrid most often wins: RAG for facts; a fine‑tuned policy for style/formatting and tool‑calling reliability.

## Data: What To Collect (and What To Skip)
- Format: instruction JSONL. Each record: {"instruction": ..., "input": ..., "output": ...}. Keep fields consistent; avoid hidden conventions.
- Quality over size: 3k–30k examples are enough for many product features. Favor diverse prompts and edge cases over near‑duplicates.
- Balance: ensure class balance (e.g., success vs. refusal), include counter‑examples, and cover hard negatives.
- Deduplicate aggressively: near‑duplicate prompts or outputs make training look good while hurting generalization.
- Fixed dev set: hold back 5–10% for evaluation; never train on it.

Example record:
```json
{"instruction": "Convert the spec to valid JSON schema.", "input": "type: user\nfields: id:int, email:string", "output": "{\n  \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n  \"type\": \"object\",\n  \"properties\": {\n    \"id\": {\"type\": \"integer\"},\n    \"email\": {\"type\": \"string\", \"format\": \"email\"}\n  },\n  \"required\": [\"id\", \"email\"]\n}"}
```

## Adapters: LoRA vs QLoRA
- LoRA (BF16/FP16 base): Great when you have multi‑GPU memory.
- QLoRA (4‑bit NF4 base): Best single‑GPU option; trains low‑VRAM while preserving quality.
- Targets: start with attention projections (q_proj, k_proj, v_proj, o_proj) and MLP (gate_up_proj / down_proj in LLaMA‑like models). Freeze norms and embeddings.
- Sizing defaults: r=8–16, alpha=16–32, dropout=0.05–0.1. Scale up only if eval plateaus.

## Training Recipes (2026)
- Supervised fine‑tuning (SFT): Cross‑entropy on instruction data. Focus on format fidelity and task success.
- Preference optimization (DPO/KTO/ORPO): If outputs must align with human preferences, add a short stage on chosen vs. rejected pairs. Keep it small (e.g., 5–20% of SFT tokens) to avoid regressions.
- Mixed‑format tasks: Train SFT with explicit format instructions + few‑shot exemplars; then optionally add a tiny DPO stage using a JSON‑valid vs. invalid discriminator to reduce format drift.

Suggested hyper‑params (tune per model):
- Optimizer: AdamW (β1=0.9, β2=0.95), weight decay 0.01 (exclude norms/biases)
- LR: 1e‑4 (LoRA) or 2e‑4 (QLoRA), cosine decay, warmup 3–5%
- Batch: effective 64–256 sequences (use grad accumulation)
- Seq length: match what you can serve; don’t train longer than your production max
- Regularization: grad clip 1.0; label smoothing 0.0–0.1 if outputs are peaky

## Evaluation: Make Failures Actionable
Track both quality and robustness:
- Formatting: exact match on JSON validity; function‑call argument validity; SQL parse success.
- Task metrics: exact match / F1 / edit distance; BLEU/ROUGE only when necessary.
- Safety/guardrails: refusal rate on disallowed prompts; jailbreak resistance on a fixed red‑team set.
- Latency and cost: tokens/sec, p95 latency, GPU‑hours per 1k requests.
- Error taxonomy: categorize failures (formatting, hallucination, refusal, tool‑call args, latency). Fix the biggest bucket first.

Tip: Maintain a “golden set” of 100–500 prompts with deterministic checkers (JSON Schema, SQL parser, unit tests). Run it on every change.

## Serving Fast: vLLM + Quantization + Speculation
- Engine: vLLM (paged attention, continuous batching) is a strong default for server inference.
- Quantization for serving: 4‑bit/8‑bit weight‑only and KV‑cache quantization can reduce memory and increase batch size. Validate quality.
- Speculative decoding: Use a small draft model or a lightweight proposer to reduce tail latency with minimal quality loss.

vLLM quickstart example:
```bash
pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8b-instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --port 8000
```

## Minimal LoRA/QLoRA (Transformers + PEFT)
```python
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model
import torch

model_id = "meta-llama/Llama-3-8b-instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# QLoRA option
# from transformers import BitsAndBytesConfig
# quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)
# model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quant, device_map="auto")

peft_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj","k_proj","v_proj","o_proj"]) 
model = get_peft_model(model, peft_cfg)

train = load_dataset("json", data_files="train.jsonl")["train"]
val = load_dataset("json", data_files="dev.jsonl")["train"]

def format_example(ex):
    prompt = f"Instruction: {ex['instruction']}\nInput: {ex.get('input','')}\nAnswer:"
    text = prompt + ex["output"]
    toks = tokenizer(text, truncation=True, max_length=2048)
    toks["labels"] = toks["input_ids"].copy()
    return toks

train_tok = train.map(format_example, remove_columns=train.column_names)
val_tok = val.map(format_example, remove_columns=val.column_names)

args = TrainingArguments(
    output_dir="out",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=16,
    learning_rate=1e-4,
    num_train_epochs=3,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    logging_steps=50,
    save_steps=1000,
    bf16=True,
)

trainer = Trainer(model=model, args=args, train_dataset=train_tok, eval_dataset=val_tok)
trainer.train()
model.save_pretrained("lora")
```

## Practical Checklists
Before training:
- [ ] Define the single metric that matters (e.g., JSON validity rate). Make it auto‑checkable.
- [ ] Curate 3–10 canonical examples per sub‑task; add edge cases.
- [ ] Decide max sequence length and the longest output you can serve.

Before shipping:
- [ ] Run the golden set; require no regression on key metrics.
- [ ] Load test the serving path (real prompts, realistic batch patterns).
- [ ] Add monitoring for p95 latency, error rate, formatting failures.

## FAQ
- Can I combine RAG and fine‑tuning? Yes—RAG for facts, a small adapter for style/format/tool calls. Most teams do both.
- Do I need RLHF? Not for most product features. SFT + a small DPO/KTO/ORPO pass often suffices.
- How many examples do I need? Start with 1–3k; grow only if eval plateaus. Always improve quality before size.

## Related
- See: /knowledge-base/rag-best-practices-2026
- See: /knowledge-base/vector-databases-for-rag
- See: /knowledge-base/speculative-decoding-inference-2026

---

Source: https://aimastermindlab.com/knowledge-base/finetuning-llms-2026 — AI Mastermind Lab (https://aimastermindlab.com)