Fine-Tuning with LoRA / QLoRA / PEFT

Fine-tuning is probably the part of this course most likely to make your head explode: QLoRA, PEFT, quantization, SFT, DPO... the jargon comes at you like a roll call.

Let me start with something reassuring — a lesson I only earned by stepping on the landmines myself: most scenarios where you think you "need fine-tuning" can actually be solved with prompting or RAG, and those options are cheaper, faster, and easier to maintain. This chapter first helps you decide whether you should fine-tune at all, then explains LoRA/QLoRA in plain language (with just enough theory and a runnable config), and finally covers how to evaluate after fine-tuning and how to avoid disasters — the step that 90% of tutorials skip, and the one where things blow up most often.

---

Don't rush to fine-tune: pick the right path first

A "the model answers our domain questions wrong" request lands on your desk, and the beginner reflex is "let's fine-tune one." Hold on — run it through this table first:

Burn this into memory: fine-tuning is good at teaching a model "how to say things" (style, format, professional voice); it is bad at stuffing in "new knowledge" (that's RAG's job). Try to fine-tune a fresh product manual into a model and most of it won't stick — and what does stick tends to come back out as hallucinations.

Honestly, I've watched too many teams demand fine-tuning on day one, burn several days of GPU time, and then discover that a well-written system prompt plus a few few-shot examples hit the target. Fine-tuning is the last resort, not the first move.

Before you touch anything, run this four-question self-check. Only proceed if every answer is "yes":

1. You've stuffed 3-5 few-shot examples into the prompt and it still doesn't hit the bar?
2. The problem is "wrong tone/format/voice," not "missing knowledge"?
3. You can assemble at least a few hundred high-quality "input → expected output" pairs?
4. You have a way to quantitatively compare results after fine-tuning (a test set, not vibes)?

If any answer is "no," go fix that one first. Don't start training.

Two real cases: one that shouldn't fine-tune, one that should

Case one (shouldn't): A cross-border e-commerce team came to me wanting to fine-tune a "customer service model that knows our products" — budget already approved. I asked one question: when the bot answers wrong, is it "doesn't know the product specs" or "says it the wrong way"? They dug through a week of support logs. Answer: the former — 90% of complaints were "irrelevant answers," because the product catalog changed every week. A product catalog is new knowledge; fine-tuning can't absorb it, and even if it could, it would be stale next week. The final solution was RAG over the product database plus a 500-word system prompt for tone. Zero GPU dollars burned, shipped in two weeks.

Case two (should): Another team doing legal documents needed to rewrite verbal descriptions into contract clauses with a fixed format. Knowledge wasn't the problem (all the templates were in the prompt), but the strong model's output had subtly different formatting every single time — punctuation, numbering, and phrasing drifted, and legal had to hand-fix every document. Eight few-shot examples still couldn't pin it down, the prompt had ballooned to 3,000 tokens, and at their volume the bill looked ugly. That combination — "knowledge is fine, format is always off, call volume is high" — is exactly fine-tuning's home turf: 2,000 human-reviewed rewrite pairs, one QLoRA run, format consistency locked in, and the system prompt shrank from 3,000 tokens to 200.

Put the two cases side by side and the decision rule is one sentence: wrong because it "doesn't know" → RAG; wrong because it "says it wrong" → fine-tune.

---

Why full fine-tuning is "unaffordable"

Term: Full Fine-Tuning

• One-line explanation: retrain all several billion parameters of the model.
• Analogy: re-educating a person from childhood just to change one catchphrase — a cannon aimed at a mosquito.
• How it's used at work: the ceiling on quality is high, but individuals and small teams basically can't afford it.
• Most common pitfall: VRAM. Full fine-tuning a 7B model needs, on top of the model itself, optimizer state (Adam stores two extra copies per parameter) plus gradients — tens to over a hundred GB of VRAM. A consumer 24G RTX 4090? Don't even think about it.

Do the rough VRAM math for full fine-tuning a 7B model and you'll see why it scares people off:

Model weights (fp16)    :  7B × 2 bytes ≈ 14 GB
Gradients (fp16)        :  7B × 2 bytes ≈ 14 GB
Adam optimizer state    :  7B × 8 bytes ≈ 56 GB   ← the big one
─────────────────────────────────────────
Total               ≈ 84 GB+  → you need A100/H100 class hardware

Hence PEFT.

Term: PEFT (Parameter-Efficient Fine-Tuning)

• One-line explanation: freeze the original model and train only a small set of extra parameters.
• Analogy: don't reinstall the whole operating system — just install a small plugin. The core is untouched, but the behavior changes.
• How it's used at work: LoRA is the most popular PEFT method; fine-tuning on a personal GPU almost always goes this route.
• Most common pitfall: assuming PEFT must be worse than full fine-tuning. On most tasks LoRA gets close enough to full fine-tuning that the gap isn't worth the cost.

---

LoRA: train only a small "patch" (the theory)

Term: LoRA (Low-Rank Adaptation)

• One-line explanation: freeze the original weights and hang a pair of small matrices alongside them to learn the "delta" — not a single original parameter changes.
• Analogy: sticking sticky-note annotations into a thick textbook. The book itself is unchanged, but read it with the notes and you reach different conclusions.
• How it's used at work: the trained "patch" (adapter) is only a few MB to a few hundred MB; one base model can carry multiple task-specific patches and swap them at will.
• Most common pitfall: setting the two knobs rank (r) and alpha blindly. Too small and it can't learn; too big wastes resources and risks overfitting.

The theory really does fit in one sentence. Fine-tuning fundamentally learns a "delta" ΔW for the original weight matrix W, so that W_new = W + ΔW. In full fine-tuning, ΔW is as big as W (billions of numbers). LoRA bets on one thing: this ΔW is actually "flat" — it can be approximated by the product of two skinny matrices:

ΔW  ≈  B × A
       │   └─ A: shape [r, original dim]   (skinny)
       └───── B: shape [original dim, r]   (tall)

Before: train an entire d×d block (e.g. 4096×4096 ≈ 16.78M numbers)
After:  train only B and A (with r=16, ≈ 4096×16×2 ≈ 131K numbers)
→ trainable parameters drop to roughly 0.8%

That r (rank) is the degree of "skinniness": smaller r means a smaller, cheaper patch with weaker expressive power. The original model stays frozen; only B and A learn — which is why both VRAM and storage plummet.

(The W here is exactly those Q/K/V projection matrices from the Transformer attention layer — if Q/K/V still feels fuzzy, read that chapter first and target_modules below will suddenly make sense.)

The two parameters you will definitely end up tuning:

---

QLoRA: squeezing a 7B model into a single 24G card

LoRA already saves on trainable parameters, but the base model itself still hogs VRAM when loaded (a 7B in fp16 is 14GB on its own). QLoRA takes one more swing.

Term: Quantization

• One-line explanation: store model parameters at lower precision (4-bit instead of 16-bit) to save VRAM.
• Analogy: compressing an HD movie down to phone-storage quality. Picture quality drops a little; file size drops a lot.
• How it's used at work: 4-bit quantization cuts the model's VRAM footprint to roughly 1/4 (14GB → about 3.5–4GB).
• Most common pitfall: quantization loses some precision. For chat/writing tasks it's basically imperceptible, but for numerically precise tasks, benchmark first — don't force it.

Term: QLoRA

• One-line explanation: first 4-bit quantize the base model to fit it into VRAM, then train a LoRA patch on top.
• Analogy: disassemble the big furniture to fit it into a small room (quantization), then renovate with sticky notes inside (LoRA).
• How it's used at work: it turned "fine-tune 7B~13B on a single consumer GPU" from fantasy into reality — the mainstream option for personal fine-tuning today.
• Most common pitfall: the 4-bit NF4 data type and double quantization settings — just copy a mature script (Unsloth / official PEFT examples). Don't improvise here.

Nail down the relationship between the three terms:

PEFT  = the umbrella idea: train only a small set of parameters
  └─ LoRA   = PEFT's concrete method: attach a low-rank "patch" (B×A)
       └─ QLoRA = LoRA + 4-bit quantized base: saves another big chunk of VRAM

Practical VRAM tiers (7B, varies with sequence length / batch size):

---

Clear the minefield first: pin your versions, never install blind

Before any code, an embarrassing story. The first time I ran QLoRA, I copied a tutorial and fought ImportError for an entire evening. The tutorial wasn't wrong — it was ten months old, and the peft and trl APIs change faster than the weather. The version pitfalls in the fine-tuning ecosystem are nastier than fine-tuning itself.

Pin version ranges when you install (the set below is a mutually compatible combination — do not just pip install -U everything to latest):

pip install "transformers>=4.51,<5" \
            "peft>=0.15,<0.18" \
            "bitsandbytes>=0.45.0" \
            "trl>=0.17,<0.20" \
            "accelerate>=1.3" \
            "datasets>=3.2"

The moment it runs, immediately pip freeze > requirements.txt. Three months from now, rerunning on a new machine, you'll thank yourself.

When old tutorial code from the internet won't run, check this table first — odds are it's a version trap:

That last point deserves its own drumroll: bitsandbytes stubbornly requires Linux + NVIDIA. An Apple Silicon Mac cannot run QLoRA locally (which is why the exercises use Colab). Don't spend two hours wrestling install errors on your MacBook — that's a fight you can't win.

---

Hands-on code: a QLoRA config that actually runs

You don't need to hand-roll everything — just recognize the key lines. Here's the standard PEFT + bitsandbytes skeleton:

# For installation, see the version-pinned command above — don't install latest blindly
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
import torch

# ① 4-bit quantization config (this is the "Q" in QLoRA)
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # NF4: a 4-bit type optimized for normally-distributed weights
    bnb_4bit_use_double_quant=True,         # double quantization, saves a bit more
    bnb_4bit_compute_dtype=torch.bfloat16,  # temporarily upcast to bf16 for compute
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B", quantization_config=bnb, device_map="auto")

# ② LoRA patch config (this is the "LoRA")
lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj","k_proj","v_proj","o_proj"],  # patch the attention Q/K/V/O projections
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()
# → trainable params: ~0.8%  ← only this much is training; everything else is frozen

Walkthrough:

1. What this does: load Qwen2.5-7B in 4-bit quantization to fit small VRAM (①), then attach a LoRA patch that's only 0.8% of the parameters (②).
2. Where to look first: start with load_in_4bit=True (the VRAM-saving switch), then r=16, lora_alpha=32 (the two knobs from earlier), and finally the ratio printed by print_trainable_parameters.
3. Key judgment call: target_modules decides which layers get patched — the attention q/k/v/o projections are the most common safe choice; adding FFN layers can help but raises both VRAM usage and overfitting risk.

Typical starting hyperparameters (with TRL's SFTTrainer): learning_rate=2e-4, num_epochs=1~3, batch_size per your VRAM (use gradient accumulation to fake a larger effective batch). Don't be greedy with epochs — 2–3 rounds is usually enough; more and the model starts memorizing (overfitting).

Getting training running: the full SFTTrainer skeleton

The code above only attaches the patch — actual training needs one more piece. Note this is the post-TRL-0.12 style: hyperparameters go into SFTConfig, not directly into the trainer (the old-tutorial style throws unexpected keyword argument, as covered in the version table above):

from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

dataset = load_dataset("json", data_files="train.jsonl", split="train")
split = dataset.train_test_split(test_size=0.1)   # hold out 10% as validation — don't skip this

config = SFTConfig(
    output_dir="./qlora-out",
    num_train_epochs=2,                    # start with 2-3 epochs, don't be greedy
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,         # effective batch = 2×8 = 16
    learning_rate=2e-4,
    max_length=1024,                       # sequence length; cut this first on OOM
    logging_steps=10,                      # log loss every 10 steps, for curve-watching
    eval_strategy="epoch",                 # run the validation set at each epoch end
    save_strategy="epoch",
    bf16=True,
)
trainer = SFTTrainer(
    model=model,                           # the LoRA-patched model from the previous section
    args=config,
    train_dataset=split["train"],
    eval_dataset=split["test"],
    processing_class=tokenizer,            # the 4.46+ name — no longer `tokenizer`
)
trainer.train()
trainer.save_model("./qlora-out/adapter") # saves only the patch, a few hundred MB

Walkthrough:

1. What this does: split the data into train/validation, configure hyperparameters, hand it to SFTTrainer to run the full training loop, and save only the LoRA patch at the end.
2. Where to look first: start with train_test_split (the validation set is your fuse against memorization), then gradient_accumulation_steps (the key to faking a large effective batch when VRAM is short), and finally the directory save_model produces — it contains an adapter, not a full model.
3. Key judgment call: eval_strategy="epoch" gives you a validation loss at each epoch end. Training loss keeps dropping while validation loss starts climbing = overfitting signal — stop at the epoch with the lowest validation loss.

What to watch during training: three loss-curve failure signals

Once training starts, don't just walk away — the loss curve announces accidents in advance:

• Loss won't budge (stuck around 2-point-something, 3-point-something): nine times out of ten it's not the learning rate — it's the data. First run print_trainable_parameters() and confirm the trainable count isn't 0 (misspell a module name in target_modules and the whole model stays frozen — nothing is learning). Then check whether the data went through the right chat template (next section). Only after ruling out both should you try bumping the learning rate up a notch from 2e-4.
• Loss falls off a cliff to near 0: that's not genius arriving — that's memorization. Too little data, too much duplication, or too many epochs. The model can now recite the training set backwards and falls apart on anything new.
• Loss becomes nan: learning rate too high, or fp16 numeric overflow. Lower the learning rate, switch the compute dtype to bfloat16 — that usually rescues it.

One-line takeaway: loss not dropping → suspect the data first; loss too low → suspect memorization; loss is nan → suspect precision.

---

Data beats parameters: format and quantity

70% of fine-tuning quality comes from data quality, not hyperparameters. The most common setup is instruction tuning (SFT, Supervised Fine-Tuning) — a pile of "input → expected output" examples:

{"instruction": "Rewrite this sentence more professionally", "input": "This thing is okay I guess", "output": "The solution's overall performance meets expectations."}
{"instruction": "Rewrite this sentence more professionally", "input": "Get it done quick",   "output": "Please arrange for this to be handled promptly."}

A few iron rules (you only learn how much these hurt by stepping on them):

• Small and precise > big and messy. A few hundred high-quality, stylistically consistent examples beat tens of thousands of sloppy ones. The model learns bad habits from your data faster than good ones.
• Format must be uniform. Whatever you want the output to look like, the training data must match it character for character — punctuation and structure included.
• Hold out a validation set. Keep a small slice of data the model never sees for evaluation (why? See overfitting in ML/DL fundamentals).

And one pitfall every beginner hits — one that never announces itself in an error message — the chat template:

Term: Chat Template

• One-line explanation: the "wrapping rule" that assembles instruction/input/output into the conversation format the model saw during pretraining.
• Analogy: however well the letter is written, put it in the wrong envelope and the recipient can't open it — every model family only recognizes its own envelope.
• How it's used at work: apply templates with tokenizer.apply_chat_template(); TRL's SFTTrainer handles standard messages-format data automatically.
• Most common pitfall: Qwen and Llama templates are mutually incompatible. Train family B's model on family A's format and the loss still goes down (it always learns something), but at inference the output is incoherent — this "training looked successful" illusion is the most treacherous kind.

How much data do you need? A feel for it by task type:

What if you don't have enough real data? Don't lower the quality bar to pad the count — generating with a strong model and filtering by hand is often more reliable than force-collected real data. Full playbook in synthetic data & augmentation.

---

⚠️ Don't ship right after fine-tuning: evaluation + catastrophic forgetting

This is the step most often skipped and most likely to blow up.

Term: Catastrophic Forgetting

• One-line explanation: in learning the new task, the model forgets general abilities it used to have.
• Analogy: drilling one difficult dance move so hard you forget how to walk.
• How it's used at work: after fine-tuning, always test both the new task and the old general abilities — don't celebrate just because the new-task score went up.
• Most common pitfall: validating only on your own fine-tuning data, declaring it "perfect!", then discovering in production that the model can now do exactly one thing and has regressed at everything else. LoRA touches only a small parameter subset, so it forgets less than full fine-tuning — but not zero.

Post-fine-tuning evaluation needs at least three legs:

1. Task set: an independent test set for your target task (unseen by the model) — did it actually learn?
2. Regression set: a batch of general questions (common sense, other formats), compared before vs. after fine-tuning — did anything degrade? (This is your forgetting alarm.)
3. Human spot checks: however high the automated score, have human eyes scan a batch of real outputs — plenty of "wrong tone / making things up" failures are invisible to automated metrics.

How to turn those three legs into a repeatable eval pipeline (instead of manually poking at it each time) is covered end-to-end in evaluation & quality monitoring — fine-tuners need it sooner or later.

One more advanced term worth recognizing — it will come up in live sessions: SFT → DPO / RLHF is a chain. SFT teaches the model "how to speak" (learning from examples); DPO/RLHF then uses preference data ("do humans prefer A or B?") to teach it "how to speak more pleasingly." For personal projects, SFT (+LoRA/QLoRA) is usually enough; DPO is for when you want further taste alignment.

---

What to do with the trained patch: mount it or merge it

Training ends and you now own an adapter directory of a few dozen to a few hundred MB. There are two roads to production, and picking the wrong one digs you a hole:

Route A: keep base and patch separate, mount at load time. At inference, load the base model first, then attach the adapter. The upside: one base can serve multiple tasks simultaneously — one patch for the support-agent voice, another for the legal voice, switched per request; inference engines like vLLM support multi-LoRA serving natively, and the VRAM savings for multi-task setups are delightful. The cost: one extra matrix addition at inference (usually imperceptible) and one more artifact to manage in your deployment pipeline.

Route B: merge_and_unload() into a single model. Weld the patch into the base weights, producing an ordinary model you can then deploy, quantize, and distribute like any open-weight model. Simplest to ship — but merging is one-way. Want to revise the patch afterwards? Your only option is retraining from scratch.

My honest recommendation: many tasks, still iterating → mount; a single task, simplest possible deployment → merge. Either way, back up the adapter directory before merging — it's only a few hundred MB, and losing it is the kind of thing you cry about.

For running the merged model locally and choosing an inference engine, continue with open-weight models, local deployment & model routing; for the compute economics in production, see deployment & cost optimization.

A quick bit of math: when fine-tuning actually saves money

The selection table at the top claimed fine-tuning can "save tokens." That claim deserves a worked example, because it's the real motivation behind many teams' fine-tuning projects:

Without fine-tuning: a 3,000-token system prompt (rules + 8 few-shot examples)
        × 100K calls per day
        = 300M input tokens per day spent on "repeating the rules"

After fine-tuning: rules and format are welded into the weights, prompt cut to 200 tokens
        Saves ~280M input tokens of spend per day
        Cost = one-time training (a few dozen dollars of QLoRA GPU rental) + self-hosted inference

But note two preconditions — miss either and the math collapses:

• Call volume must be large. At a few hundred calls a day, the token savings might take years to pay back the training + self-hosting costs.
• Self-hosting has its own bill. An open-weight model plus patch means babysitting GPUs yourself — how to estimate that cost is covered in full in deployment & cost optimization.

One-line takeaway: fine-tuning saves money only when "volume is high AND the prompt is long" — missing either, don't use cost as your justification.

---

⚠️ Common pitfalls

---

🔧 Hands-on exercises

Exercise 1: Selection judgment + hand-writing SFT data (about 20 minutes, no GPU needed)

1. Give yourself a real requirement (e.g., "rewrite casual support-agent replies into formal written language"), then judge against the three-way table at the top: Prompt, RAG, or fine-tuning? Write one sentence of reasoning.
2. Assuming the verdict is fine-tuning, hand-write 10 SFT examples for the task in the JSON format above, with completely uniform style, punctuation, and structure.
3. When done, randomly pull 2 of the 10 aside as a "validation set" you're not allowed to touch — that's your first evaluation habit.

Acceptance criteria: around example 6 or 7 your head starts hurting and you start thinking "maybe I'll have AI generate a few" — congratulations, you've just experienced the industry consensus that "assembling high-quality data is the most exhausting part of fine-tuning."

Exercise 2: Actually run QLoRA once on Unsloth Colab (about 40–60 minutes, free GPU)

1. Open the official Unsloth GitHub and find a free Colab notebook (pick a Qwen or Llama 4-bit example) — a free Colab T4 is enough. Don't fight bitsandbytes on a local Mac; as covered earlier, that's a fight you can't win.
2. Match every parameter against this chapter line by line: load_in_4bit, r, lora_alpha, learning_rate, num_train_epochs — say out loud what each one controls before you change it.
3. After the default example runs, swap the training data for your 10 hand-written examples from Exercise 1 (mind the chat template), and watch the loss curve — with data this small, you'll very likely see the "loss dropping too fast" memorization signal, a live demo of the failure signals covered above.
4. After training, run the double test: first ask it your target task, then ask a general question like "what's 1+1". It only counts as a pass if both sides answer well — that's the minimal version of "task set + regression set" evaluation.

---

📚 Key takeaways

1. Fine-tuning is the last resort: if Prompt / RAG works, don't touch it. Fine-tuning handles "how to say it" (style/format), not "new knowledge" (that's RAG).
2. Full fine-tuning is a VRAM monster (80G+ for a 7B); LoRA uses the low-rank patch ΔW ≈ B×A to cut trainable parameters to ~0.8%, and the patch is only a few MB.
3. QLoRA = 4-bit quantized base + LoRA, cutting VRAM to 6–12G — a single consumer GPU can train a 7B; start at r=16 / alpha=32, lr=2e-4, 2–3 epochs; pin your dependency versions (peft/trl/transformers APIs churn fast, and bitsandbytes only works on Linux + CUDA).
4. Data > parameters: small and precise, uniform format, the right chat template, a held-out validation set; data volume runs from hundreds to tens of thousands depending on the task.
5. Evaluation after fine-tuning is mandatory: task set (did it learn) + regression set (guard against catastrophic forgetting) + human spot checks — never ship on the new-task score alone; back up the adapter before merging; add DPO only when you want further taste alignment.

---

Related reading

• Open-weight models, local deployment & model routing — how to run your fine-tuned model locally (especially after merging) and choose an inference engine
• Transformer & attention fundamentals — what the q/k/v/o in target_modules actually are
• ML / DL fundamentals crash course — overfitting, validation sets, and other concepts fine-tuning can't avoid
• Synthetic data & augmentation — how to manufacture SFT data when you don't have enough
• RAG basics — the road that "stuffing in new knowledge" should actually take
• Evaluation & quality monitoring — turning "task set + regression set" into a repeatable eval pipeline

Next up: Deployment & cost optimization →