LoRA & QLoRA

B is initialised to zero so the adapter output is zero at the start of training — the model begins as the original pretrained model. A is randomly initialised. After training, BA can be merged into W with zero inference overhead.

LoRA with PEFT library — fine-tuning LLaMA 3 8B

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

lora_config = LoraConfig(
    r=8,                          # rank — higher = more capacity, more params
    lora_alpha=16,                # scaling: effective update = (alpha/r) * BA = 2x
    target_modules=[              # apply LoRA to these projection matrices
        "q_proj", "k_proj", "v_proj", "o_proj",   # attention
        "gate_proj", "up_proj", "down_proj",        # feedforward (MLP)
    ],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.52

# After training, merge adapter into base weights (zero inference overhead)
merged_model = model.merge_and_unload()

QLoRA: fine-tune LLaMA 3 70B on a single 48GB A100

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
import torch

# 4-bit NF4 quantisation config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # Normal Float 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16,  # compute in BF16
    bnb_4bit_use_double_quant=True,     # double quantisation
)

# Load 70B model quantised to 4-bit (~35GB instead of 140GB)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B",
    quantization_config=bnb_config,
    device_map="auto",
)

# Add LoRA adapters in BF16 on top of frozen 4-bit base
lora_config = LoraConfig(r=16, lora_alpha=32,
                         target_modules=["q_proj","v_proj","k_proj","o_proj"],
                         lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(model, lora_config)
# Peak GPU memory: ~40GB — fits on a single 48GB A100 or 4× 12GB consumer GPUs

1. What is the mathematical justification for LoRA's low-rank assumption? (Answer: The hypothesis: weight updates ΔW = W_finetuned - W_pretrained have low intrinsic rank. Empirical evidence (Aghajanyan et al. 2020): fine-tuned models can be represented in low-dimensional intrinsic subspaces. The pretrained model already captures most of the relevant structure; task-specific adaptation requires a low-rank modification. LoRA decomposes ΔW = BA where B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k}, r << min(d,k). Hypothesis confirmed: r=4–16 achieves near full-rank fine-tuning quality on most NLP tasks.)
2. How does LoRA merge adapters at inference time for zero overhead? (Answer: LoRA adds ΔW = BA to the frozen W₀. At inference: output = (W₀ + BA)x = W₀x + BAx. Pre-merging: must compute both W₀x and BAx, then add — two matrix multiplications. Post-merging: compute W_merged = W₀ + BA (one-time operation), then output = W_merged × x — single matrix multiplication, identical speed to original model. Merging is a pre-deployment step: load LoRA weights, add to base weights, save. The merged model is indistinguishable from full fine-tuning at serving time.)
3. What is the effect of LoRA rank r on model capacity and training stability? (Answer: Low r (1–4): very few trainable parameters (~100K for 7B model). Fast, memory-efficient, less risk of overfitting. May underfit complex tasks. Medium r (8–32): standard range for most fine-tuning. Balances capacity and efficiency. High r (64–256): approaches full fine-tuning capacity. More parameters but still much cheaper than full FT. Stability: very high r can cause instability if learning rate is not reduced proportionally. Practical guideline: start with r=16, tune if underfitting/overfitting observed.)
4. What is the difference between applying LoRA to attention weights only vs all linear layers? (Answer: Attention-only LoRA (q_proj, v_proj, k_proj, o_proj): targets the self-attention mechanism — where most task-specific information integration happens. Fewer parameters, faster. Original LoRA paper used q, v projections only. All-linear LoRA (attention + MLP + embeddings): more capacity to adapt. Usually better accuracy on complex tasks requiring deep factual changes. Memory cost: ~4× more LoRA parameters. For instruction following and style: attention-only sufficient. For domain knowledge adaptation (medical, legal): all-linear LoRA recommended.)
5. What is QLoRA and how does it enable 65B model fine-tuning on a single GPU? (Answer: QLoRA (Dettmers et al. 2023): (1) Load base model in 4-bit NF4 (Normal Float 4) quantisation — 65B model: ~35GB instead of ~130GB. (2) Add LoRA adapters in BF16. (3) Train only LoRA adapters (base model frozen and quantised). (4) During forward/backward pass: dequantise base weights to BF16 on the fly for computation, then discard. Memory: 35GB (quantised base) + 4GB (LoRA + optimizer states) ≈ 39GB — fits on a single 40GB A100. Quantisation adds <1% performance loss vs FP16 LoRA on most tasks.)