Fine-Tuning Llama 3 on Custom Data (Step-by-Step)

TL;DR: While RAG is great for injecting facts, fine-tuning is necessary to teach a model a specific behavior, tone, or complex output format (like generating strict JSON or proprietary code syntax). This guide shows you how to fine-tune Llama 3 8B using QLoRA and Unsloth on a single cloud GPU in under an hour.

In 2026, you rarely need to perform "full" fine-tuning (updating all 8 billion parameters). Instead, we use QLoRA (Quantized Low-Rank Adaptation). We freeze the base model in 4-bit precision, and only train a tiny "adapter" network on top of it. This reduces VRAM requirements from 40GB+ down to just 8GB, making it accessible on cheap GPUs.

We will use Unsloth, an incredibly optimized library that makes fine-tuning Llama models 2x faster and uses 70% less memory than standard Hugging Face tools.

Prerequisites

• A GPU instance (e.g., RunPod, Google Colab Pro, vast.ai) with an NVIDIA GPU (RTX 3090, A10g, A100)
• Python 3.10+

Install Unsloth and dependencies:

pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
pip install --no-deps "trl<0.9.0" peft accelerate bitsandbytes

Step 1: The Dataset Format

The most critical part of fine-tuning is your data. You need 500 to 2,000 high-quality examples of how you want the model to behave.

The standard format for instruction fine-tuning is the Alpaca format, usually stored in a JSONL file:

{
  "instruction": "You are a support bot. Write a polite apology for a delayed order.",
  "input": "Order #12345 (delayed by 3 days)",
  "output": "I am so sorry to hear that Order #12345 has been delayed by 3 days. We are upgrading your shipping to overnight immediately."
}

Step 2: Load the Model with Unsloth

We will load the base Llama-3-8B model directly in 4-bit precision.

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048 # Define how long your inputs/outputs will be

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/llama-3-8b-bnb-4bit", # The pre-quantized unsloth version
    max_seq_length = max_seq_length,
    dtype = None, 
    load_in_4bit = True,
)

Step 3: Add the LoRA Adapters

Now we add the trainable adapter layers.

model = FastLanguageModel.get_peft_model(
    model,
    r = 16, # Rank: How expressive the adapter is (usually 8, 16, or 32)
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", 
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0, # Unsloth optimizes dropout to 0
    bias = "none",    
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

Step 4: Formatting the Dataset

LLMs need text to be formatted with specific special tokens so they know when an instruction ends and a response begins. Llama 3 uses specific <|start_header_id|> tokens, but Unsloth provides a helper template to map our Alpaca JSON data into the exact format Llama 3 expects.

from datasets import load_dataset

alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}<|eot_id|>"""

EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN

def formatting_prompts_func(examples):
    instructions = examples["instruction"]
    inputs       = examples["input"]
    outputs      = examples["output"]
    texts = []
    for instruction, input, output in zip(instructions, inputs, outputs):
        # Must add EOS_TOKEN, otherwise generation goes on forever!
        text = alpaca_prompt.format(instruction, input, output) + EOS_TOKEN
        texts.append(text)
    return { "text" : texts, }

# Load your json dataset
dataset = load_dataset("json", data_files="my_dataset.json", split = "train")

# Apply the formatting
dataset = dataset.map(formatting_prompts_func, batched = True,)

Step 5: Start Training!

We use the SFTTrainer (Supervised Fine-Tuning) from Hugging Face's trl library to execute the training loop.

from trl import SFTTrainer
from transformers import TrainingArguments
import torch

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    dataset_num_proc = 2,
    packing = False, # Can make training 5x faster for short sequences!
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60, # Tweak this based on dataset size (num_epochs = 1 or 2 is usually enough)
        learning_rate = 2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        output_dir = "outputs",
    ),
)

trainer_stats = trainer.train()

On an RTX 3090/A10g or an A100, a dataset of 1,000 examples takes less than 15 minutes to fine-tune using Unsloth.

Step 6: Test the Fine-Tuned Model

Once training is done, test inference locally in your script:

# Switch to fast inference mode
FastLanguageModel.for_inference(model) 

inputs = tokenizer(
[
    alpaca_prompt.format(
        "You are a support bot. Write a polite apology for a delayed order.", # instruction
        "Order #999 (delayed 1 week)", # input
        "", # output - leave this blank for generation!
    )
], return_tensors = "pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens = 64, use_cache = True)
print(tokenizer.batch_decode(outputs))

Step 7: Saving the Adapter (and GGUF for Ollama)

The brilliant part of LoRA is that your trained weights are tiny (~150MB). You can save them locally or push them to the Hugging Face hub.

# Save the tiny LoRA adapter locally
model.save_pretrained("lora_adapter")

# Or, merge it with the base model and export directly to GGUF (for use in Ollama/LM Studio!)
model.save_pretrained_gguf("lora_gguf_model", tokenizer, quantization_method = "q4_k_m")

If you exported to GGUF, you can easily load your new custom model in Ollama using a Modelfile, allowing you to run your enterprise-specific Llama 3 model entirely offline.