Fine‑Tuning LLMs in 2026: LoRA, QLoRA, and vLLM

Fine‑tuning gives an LLM new habits and stronger consistency for your specific product tasks; retrieval (RAG) gives it fresh facts and citations. In 2026, most reliable systems combine both.

TL;DR

• Fine‑tune when you need new behavior, domain style, or strict output formats that prompting + RAG cannot reliably produce. Keep RAG for fresh or fast‑changing knowledge and for citations.
• Start with small, well‑curated instruction data. Use parameter‑efficient adapters (LoRA/QLoRA). Measure formatting accuracy and task quality, not just loss.
• Serve with an inference engine that supports paged/segmented KV‑cache and continuous batching (e.g., vLLM). Add speculative decoding to cut tail latency.

When To Fine‑Tune vs. Use RAG

Use this quick decision guide:

• Need up‑to‑date facts, references, or long‑tail entities → Prefer RAG. Add strong retrieval + re‑ranking and cite sources.
• Need consistent structure (JSON, SQL, DSL), brand tone, or task‑specific steps → Fine‑tune a small adapter on examples showing exactly the target behavior.
• Hybrid most often wins: RAG for facts; a fine‑tuned policy for style/formatting and tool‑calling reliability.

Data: What To Collect (and What To Skip)

• Format: instruction JSONL. Each record: {"instruction": ..., "input": ..., "output": ...}. Keep fields consistent; avoid hidden conventions.
• Quality over size: 3k–30k examples are enough for many product features. Favor diverse prompts and edge cases over near‑duplicates.
• Balance: ensure class balance (e.g., success vs. refusal), include counter‑examples, and cover hard negatives.
• Deduplicate aggressively: near‑duplicate prompts or outputs make training look good while hurting generalization.
• Fixed dev set: hold back 5–10% for evaluation; never train on it.

Example record:

{"instruction": "Convert the spec to valid JSON schema.", "input": "type: user\nfields: id:int, email:string", "output": "{\n  \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n  \"type\": \"object\",\n  \"properties\": {\n    \"id\": {\"type\": \"integer\"},\n    \"email\": {\"type\": \"string\", \"format\": \"email\"}\n  },\n  \"required\": [\"id\", \"email\"]\n}"}

Adapters: LoRA vs QLoRA

• LoRA (BF16/FP16 base): Great when you have multi‑GPU memory.
• QLoRA (4‑bit NF4 base): Best single‑GPU option; trains low‑VRAM while preserving quality.
• Targets: start with attention projections (q_proj, k_proj, v_proj, o_proj) and MLP (gate_up_proj / down_proj in LLaMA‑like models). Freeze norms and embeddings.
• Sizing defaults: r=8–16, alpha=16–32, dropout=0.05–0.1. Scale up only if eval plateaus.

Training Recipes (2026)

• Supervised fine‑tuning (SFT): Cross‑entropy on instruction data. Focus on format fidelity and task success.
• Preference optimization (DPO/KTO/ORPO): If outputs must align with human preferences, add a short stage on chosen vs. rejected pairs. Keep it small (e.g., 5–20% of SFT tokens) to avoid regressions.
• Mixed‑format tasks: Train SFT with explicit format instructions + few‑shot exemplars; then optionally add a tiny DPO stage using a JSON‑valid vs. invalid discriminator to reduce format drift.

Suggested hyper‑params (tune per model):

• Optimizer: AdamW (β1=0.9, β2=0.95), weight decay 0.01 (exclude norms/biases)
• LR: 1e‑4 (LoRA) or 2e‑4 (QLoRA), cosine decay, warmup 3–5%
• Batch: effective 64–256 sequences (use grad accumulation)
• Seq length: match what you can serve; don’t train longer than your production max
• Regularization: grad clip 1.0; label smoothing 0.0–0.1 if outputs are peaky

Evaluation: Make Failures Actionable

Track both quality and robustness:

• Formatting: exact match on JSON validity; function‑call argument validity; SQL parse success.
• Task metrics: exact match / F1 / edit distance; BLEU/ROUGE only when necessary.
• Safety/guardrails: refusal rate on disallowed prompts; jailbreak resistance on a fixed red‑team set.
• Latency and cost: tokens/sec, p95 latency, GPU‑hours per 1k requests.
• Error taxonomy: categorize failures (formatting, hallucination, refusal, tool‑call args, latency). Fix the biggest bucket first.

Tip: Maintain a “golden set” of 100–500 prompts with deterministic checkers (JSON Schema, SQL parser, unit tests). Run it on every change.

Serving Fast: vLLM + Quantization + Speculation

• Engine: vLLM (paged attention, continuous batching) is a strong default for server inference.
• Quantization for serving: 4‑bit/8‑bit weight‑only and KV‑cache quantization can reduce memory and increase batch size. Validate quality.
• Speculative decoding: Use a small draft model or a lightweight proposer to reduce tail latency with minimal quality loss.

vLLM quickstart example:

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8b-instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --port 8000

Minimal LoRA/QLoRA (Transformers + PEFT)

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model
import torch

model_id = "meta-llama/Llama-3-8b-instruct"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

# QLoRA option
# from transformers import BitsAndBytesConfig
# quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)
# model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quant, device_map="auto")

peft_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj","k_proj","v_proj","o_proj"]) 
model = get_peft_model(model, peft_cfg)

train = load_dataset("json", data_files="train.jsonl")["train"]
val = load_dataset("json", data_files="dev.jsonl")["train"]

def format_example(ex):
    prompt = f"Instruction: {ex['instruction']}\nInput: {ex.get('input','')}\nAnswer:"
    text = prompt + ex["output"]
    toks = tokenizer(text, truncation=True, max_length=2048)
    toks["labels"] = toks["input_ids"].copy()
    return toks

train_tok = train.map(format_example, remove_columns=train.column_names)
val_tok = val.map(format_example, remove_columns=val.column_names)

args = TrainingArguments(
    output_dir="out",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=16,
    learning_rate=1e-4,
    num_train_epochs=3,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    logging_steps=50,
    save_steps=1000,
    bf16=True,
)

trainer = Trainer(model=model, args=args, train_dataset=train_tok, eval_dataset=val_tok)
trainer.train()
model.save_pretrained("lora")

Practical Checklists

Before training:

• Define the single metric that matters (e.g., JSON validity rate). Make it auto‑checkable.
• Curate 3–10 canonical examples per sub‑task; add edge cases.
• Decide max sequence length and the longest output you can serve.

Before shipping:

• Run the golden set; require no regression on key metrics.
• Load test the serving path (real prompts, realistic batch patterns).
• Add monitoring for p95 latency, error rate, formatting failures.

FAQ

• Can I combine RAG and fine‑tuning? Yes—RAG for facts, a small adapter for style/format/tool calls. Most teams do both.
• Do I need RLHF? Not for most product features. SFT + a small DPO/KTO/ORPO pass often suffices.
• How many examples do I need? Start with 1–3k; grow only if eval plateaus. Always improve quality before size.

Related

• See: /knowledge-base/rag-best-practices-2026
• See: /knowledge-base/vector-databases-for-rag
• See: /knowledge-base/speculative-decoding-inference-2026