The Complete Guide to LLM Fine-Tuning: QLoRA, DPO, and Domain Adaptation

# The Complete Guide to LLM Fine-Tuning: QLoRA, DPO, and Domain Adaptation

While general-purpose LLMs excel at conversational tasks, enterprise applications often demand specialized terminology, strict JSON output formatting, or alignment with internal compliance guidelines. In these cases, fine-tuning an open-weights model (such as Meta Llama 3 or Mistral) or a closed API model provides dramatic performance and latency advantages.

---

When to Fine-Tune vs RAG

| Criteria | RAG | Fine-Tuning | | --- | --- | --- | | Dynamic/Frequently Changing Knowledge | Excellent | Poor (Requires retraining) | | Specialized Output Tone / Style | Moderate | Superior | | Enforcing Strict JSON Schema Output | Moderate | Superior | | Reducing Cost & Latency | Moderate | Superior (Use smaller 7B/8B model) | | Data Privacy / On-Premise | Good | Superior (Full weight ownership) |

---

The QLoRA Fine-Tuning Pipeline

Parameter-Efficient Fine-Tuning (PEFT) with **QLoRA** (Quantized Low-Rank Adaptation) enables fine-tuning 70B parameter models on accessible GPU hardware (e.g., single NVIDIA A100/H100) without sacrificing model capability.

Step 1: Dataset Curation & Formatting Quality vastly outperforms quantity in fine-tuning. 1,000 highly curated, verified instruction-response pairs yield better results than 100,000 noisy samples.

Step 2: Training Configuration - **Quantization:** 4-bit NormalFloat (NF4) - **LoRA Rank (r):** 16 or 32 for general tasks; 64 for complex coding/reasoning - **LoRA Alpha:** 32 or 64 (typically 2 × rank) - **Target Modules:** Query, Key, Value, Output projections (`q_proj, k_proj, v_proj, o_proj`)

Step 3: Preference Alignment with DPO (Direct Preference Optimization) After Supervised Fine-Tuning (SFT), apply DPO using pairs of preferred vs dispreferred responses to align the model with human preferences without the complexity of traditional RLHF reward models.

---

Conclusion By combining 4-bit QLoRA with DPO alignment, enterprises can train proprietary 8B or 70B parameter LLMs that rival proprietary cloud APIs at a fraction of the operating cost.