[Local LLM 28] Understand LoRA, QLoRA and a Small Training Script

Building a Local LLM · 28/30 · Advanced and Operations

Full series and learning sequence · Practice code ZIP

For new document knowledge, first consider whether retrieval can supply the information. Fine-tuning is a separate option when the goal is to change a repeated response pattern or classification behavior. This part explains LoRA and QLoRA and follows the structure of a small training script.

The optional training exercise needs CUDA-capable WSL/Linux and a separate Python environment. Keep it separate from the virtual environment used for the Windows API exercises.

Learning the amount of change in LoRA·QLoRA

LoRA freezes the existing weights W and represents their update as the product of two small matrices, ΔW=BA. Instead of training all d×k parameters, it trains r(d+k) parameters at rank r. This reduces the trainable parameter count, but a small r may not represent every task adequately.

QLoRA freezes a quantized base model and trains LoRA adapters. The paper uses techniques including NF4 and double quantization. This does not mean choosing an arbitrary 4-bit inference file and training it directly. The example follows the paths supported by Transformers, PEFT and bitsandbytes.

Separate learning materials and assessments

For questions that require facts from source documents, start with RAG in Part 17. Consider fine-tuning when the requirement concerns repeated behavior such as output format, tone or classification. Use training data only after checking privacy, copyright and permission to use it for training.

train.jsonl contains three fictional English question-and-answer examples. The 5-step training run is a smoke test of the connections between components. It cannot demonstrate improved Korean ability or production performance. A decrease in training loss still needs validation with separate evaluation data.

Optional Lab: 5-Step Connectivity Check in CUDA

The training example targets CUDA-capable WSL/Linux with Python 3.12. Do not reuse the existing Windows API virtual environment. Install the appropriate CUDA package using the official PyTorch installation selector, then check that the following command detects the GPU.

python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
python -m pip install transformers peft accelerate bitsandbytes datasets

Place train_qlora.py and train.jsonl in the same folder. Memory requirements depend on the GPU, package versions and sequence length. The checks recorded for this example cover syntax and data flow. Training on CUDA hardware requires separate verification; no specific VRAM capacity is guaranteed to succeed.

import json
from pathlib import Path
import torch
from datasets import Dataset
from transformers import (AutoTokenizer, AutoModelForCausalLM,
    BitsAndBytesConfig, TrainingArguments, Trainer, DataCollatorForSeq2Seq)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

if not torch.cuda.is_available():
    raise SystemExit("This optional example requires a CUDA-capable PyTorch environment.")
base = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
compute = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
quant = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=compute)
tokenizer = AutoTokenizer.from_pretrained(base)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(base,
    quantization_config=quant, device_map={"": 0}, dtype=compute)
model.config.use_cache = False
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, LoraConfig(r=8, lora_alpha=16,
    target_modules=["q_proj", "v_proj"], lora_dropout=0.05,
    bias="none", task_type="CAUSAL_LM"))
rows = [json.loads(line) for line in
    Path(__file__).with_name("train.jsonl").read_text(encoding="utf-8").splitlines() if line.strip()]
def encode(row):
    messages = row["messages"]
    prefix = tokenizer.apply_chat_template(messages[:-1], tokenize=False,
                                           add_generation_prompt=True)
    full = tokenizer.apply_chat_template(messages, tokenize=False)
    result = tokenizer(full, add_special_tokens=False, truncation=True, max_length=256)
    n = len(tokenizer(prefix, add_special_tokens=False)["input_ids"])
    result["labels"] = [-100] * min(n, len(result["input_ids"])) + result["input_ids"][n:]
    if all(x == -100 for x in result["labels"]):
        raise ValueError("No answer tokens left after truncation")
    return result
original = Dataset.from_list(rows)
data = original.map(encode, remove_columns=original.column_names)
args = TrainingArguments(output_dir="adapter-smoke", max_steps=5,
    per_device_train_batch_size=1, gradient_accumulation_steps=1,
    learning_rate=2e-4, gradient_checkpointing=True, logging_steps=1,
    save_strategy="no", report_to="none", bf16=compute == torch.bfloat16,
    fp16=compute == torch.float16, optim="paged_adamw_8bit")
model.print_trainable_parameters()
Trainer(model=model, args=args, train_dataset=data,
    data_collator=DataCollatorForSeq2Seq(tokenizer, padding=True,
                                       label_pad_token_id=-100)).train()
model.save_pretrained("adapter-smoke")
tokenizer.save_pretrained("adapter-smoke")

python train_qlora.py
python -m pip freeze > training-requirements.txt

Adapters and pre- and post-learning evaluations

adapter-smoke stores the adapter and tokenizer configuration. These files do not replace the base model. Inference needs a loading path compatible with the same base model. GGUF conversion and quantization are separate steps, with support that depends on the architecture and adapter.

Use the same evaluation questions before and after fine-tuning. Check whether the model follows the new format at the expense of general questions, and whether it reproduces training sentences. If unseen examples do not improve, revisit the dataset and task definition before extending training time.

reference material

• LoRA original paper
• QLoRA original paper
• PEFT Quantization Learning
• Transformers Trainer
• PyTorch official installation

Documentation and model tags checked: October 18, 2026. Primary lab environment: Separate learning environment on WSL/Linux with CUDA enabled.

---