Fine-tune a model with LoRA on a single GPU

Fine-tuning a 7-billion-parameter model used to mean a multi-GPU cluster and a cloud bill. LoRA changed the math: you freeze the base model, train a tiny pair of low-rank matrices alongside each frozen weight, and adapt a modern LLM on a single GPU — sometimes a consumer one. This tutorial walks through the whole evening: picking a model, preparing data, training, and evaluating.

Why LoRA (and why QLoRA)#

Full fine-tuning updates every weight. For a 7B model, Adam's optimizer states alone push memory well past 80 GB in BF16. LoRA, introduced by Hu et al. at Microsoft (arXiv:2106.09685, June 2021), rests on one observation: the updates to the weights during fine-tuning have low intrinsic rank. Instead of updating a 4096×4096 matrix (16.7M parameters), you freeze it and train two small matrices, B (4096×r) and A (r×4096), whose product approximates the update. With rank r=16, that's 131,072 parameters — about 0.8% of the original layer — and the original paper reported matching full fine-tuning quality on GPT-3 175B while cutting trainable parameters roughly 10,000×.

Training typically touches 0.1–1% of total parameters. At inference, the adapter can be merged back into the base weights, adding zero latency overhead.

QLoRA (Dettmers et al., University of Washington, arXiv:2305.14314, May 2023) goes one step further: load the frozen base model in 4-bit precision (NF4), keep the LoRA adapters in BF16, and page optimizer states to CPU when memory runs short. The paper demonstrated fine-tuning a 65B Llama on a single 48 GB GPU. In practice, a 7B model drops from ~28 GB of weights in FP16 to roughly 4–6 GB, making a 16–24 GB consumer card plenty.

Pick QLoRA if you're on a single consumer GPU; plain LoRA if you have 24 GB+ and want maximum quality.

1. Pick your base model and environment#

Use a small instruct model for your first run — Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3 are common, ungated choices; gated weights like Llama 3.1 require a Hugging Face access request. You need Python 3.10+, PyTorch with CUDA, and one GPU with at least 16 GB VRAM (8 GB can work for a 7B QLoRA run with short sequences).

pip install torch transformers peft bitsandbytes trl datasets

Note: pin versions in a requirements.txt. The PEFT and bitsandbytes APIs move fast — community guides consistently warn that unpinned installs are the most common cause of broken tutorials.

2. Prepare your data#

Data quality matters more than any hyperparameter. LoRA adapts the model to your domain; it cannot invent knowledge you don't feed it. Follow a simple pipeline:

1. Define the task narrowly. Support-ticket triage, code documentation style, medical-terminology translation — one job, not five.
2. Format as instruction pairs. Use chat-template JSONL with instruction, input, and output fields, or the Alpaca format.
3. Deduplicate and filter. A few thousand clean examples beat 100k noisy ones. Aim for roughly 1,000–10,000 high-quality examples for a first LoRA run.
4. Split off a holdout set. You need ~100–200 prompts you'll never train on, for evaluation later. This is the step everyone skips and regrets.

{"instruction": "Classify the support ticket into billing, technical, or account.", "input": "I was charged twice for my subscription.", "output": "billing"}

Keep sequences short enough for your VRAM: max_seq_length of 2048 is a safe default on consumer cards.

3. Configure the adapters#

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
tokenizer.pad_token = tokenizer.eos_token

lora_config = LoraConfig(
    r=16,                       # rank: 8–32 typical
    lora_alpha=32,              # often 2x the rank
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # expect ~0.2–0.3% trainable for 7B

Key knobs, and sane starting values:

• Rank (r): 16 is the workhorse. Higher ranks (32–64) help on complex tasks but increase memory and overfit risk.
• Alpha: commonly set to 2× rank; it scales the adapter's contribution independently of the base learning rate.
• Target modules: attention projections (q_proj, k_proj, v_proj, o_proj) at minimum; adding the MLP projections (gate/up/down_proj) usually helps on harder tasks.
• Learning rate: 1e-4 to 3e-4 — notably higher than the 1e-5 to 5e-5 typical for full fine-tuning, because only a few parameters are moving. 2e-4 is the standard starting point.
• Epochs: 2–4. Fine-tuning overfits fast; watch validation loss, not just training loss.

4. Train on a single GPU#

Use the TRL library's SFTTrainer, which wraps the Hugging Face Trainer with supervised fine-tuning defaults:

from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

dataset = load_dataset("json", data_files="data/train.jsonl", split="train")

args = SFTConfig(
    output_dir="./checkpoints/lora_run",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,   # effective batch = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    bf16=True,
    gradient_checkpointing=True,
    max_seq_length=2048,
    logging_steps=25,
    eval_strategy="steps",
    eval_steps=200,
    save_steps=200,
    max_grad_norm=0.3,
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=args,
)
trainer.train()

If VRAM is tight, drop per_device_train_batch_size to 1 and raise gradient_accumulation_steps — the effective batch size is what matters for stability. Gradient checkpointing trades ~20–30% slower training for substantially lower memory. On a 7B QLoRA run, expect a few hours on a single RTX 3090-class card, not days.

Watch two signals while training: training loss (should fall steadily) and eval loss on your holdout set (should fall, then eventually rise — that turn is where you stop). A run where training loss keeps dropping while eval loss climbs is memorizing your data, not learning the task.

5. Evaluate like you mean it#

Eval is where hobbyist runs and production runs diverge. Do all three of these:

1. Automatic metrics on the holdout set. For classification or structured output, exact-match accuracy or schema validity is cheap and unambiguous.
2. Side-by-side comparisons. Generate answers from the base model and the LoRA adapter on the same 50–100 prompts, and judge blind — either yourself or with a strong model as judge. Compare against a prompt-engineering baseline too; sometimes the adapter's gains are smaller than a better system prompt would have given you.
3. Regression probes. Ask general-knowledge and reasoning questions the model used to handle. LoRA can cause catastrophic forgetting when the learning rate is too high or the data is too narrow; a 20-question regression suite catches it.

Only merge and deploy when the adapter beats the base model and beats your prompt-engineering baseline. Otherwise, iterate on data, not hyperparameters.

6. Merge or serve the adapter#

You have two deployment paths:

• Merge the adapter into the base weights for a single self-contained model: model = model.merge_and_unload(). Simpler serving, zero adapter-loading code — but you lose the ability to hot-swap adapters.
• Serve base + adapter separately. vLLM and llama.cpp both support loading LoRA adapters at runtime, which lets one base model host many adapters.

Push the adapter to the Hugging Face Hub with model.push_to_hub(...) — adapters are tiny (tens of megabytes), so sharing them is nearly free.

Common pitfalls#

• Too much data, too little curation. LoRA amplifies whatever's in your dataset, including its mistakes and formatting quirks.
• Learning rate too high. Above ~5e-4 with LoRA, models tend to degrade into repetition or lose general ability.
• No eval baseline. Without a base-model comparison on the same prompts, you can't tell whether fine-tuning did anything.
• Training on your test set. Keep the holdout set strictly separate from day one.
• Expecting new knowledge. LoRA is excellent at teaching style, format, and task behavior. It is poor at teaching facts the base model never saw — that's a retrieval (RAG) problem, not a fine-tuning problem.

The takeaway#

One evening, one GPU, and a few thousand curated examples is enough to teach an open-weights model to behave like your domain expert — not by rewriting its weights, but by training a 0.3% adapter that steers them. The formula that works: narrow task, clean data, holdout eval, rank 16, LR 2e-4, 3 epochs, early-stop on eval loss. Everything else is iteration.