Personal Wiki

How to Make Your LLM More Accurate with RAG & Fine-Tuning

4 min read

How to Make Your LLM More Accurate with RAG & Fine-Tuning

Abstract

This Towards Data Science article by Sarah Schürch provides a practitioner-oriented comparison of Retrieval Augmented Generation (RAG) and fine-tuning as two complementary strategies for extending LLM capabilities beyond their training cutoff. RAG leaves model weights unchanged and injects external knowledge at inference time via a retrieval pipeline (embedding → vector search → context injection); fine-tuning permanently encodes new knowledge into model weights through supervised training on domain-specific data. The article walks through the technical mechanics of each approach, a structured comparison of their tradeoffs (flexibility, inference latency, compute cost), LangChain and Hugging Face implementation paths for RAG, LoRA/QLoRA for compute-efficient fine-tuning, and use-case heuristics for choosing between them. It closes with a brief treatment of RAFT (Retrieval Augmented Fine-Tuning), a hybrid that applies fine-tuning first for domain terminology then extends with RAG for dynamic knowledge.

Key Concepts

  • RAG (Retrieval Augmented Generation): Extends LLM inference by retrieving relevant documents from an external knowledge source and injecting them into the prompt as context. The model weights are never modified — all adaptation happens at inference time. Suited for dynamic or frequently updated content.
  • Fine-Tuning: Continues training a pre-trained LLM on domain-specific data to update its weights, internalising terminology, style, and task-specific patterns. Produces faster inference (no retrieval step) but is compute-intensive upfront and requires large, high-quality training datasets.
  • RAFT (Retrieval Augmented Fine-Tuning): Hybrid approach — fine-tune first for domain vocabulary and structure, then add RAG for real-time external knowledge access. Combines deep expertise with dynamic adaptability.
  • Query Embedding: First step of the RAG pipeline — the user query is converted to a dense vector representation (e.g. text-embedding-ada-002, all-MiniLM-L6-v2) so that semantic similarity search can be performed against the vector database.
  • Approximate Nearest Neighbors (ANN): The similarity search algorithm used within vector databases to efficiently find documents closest to the query embedding. FAISS (Meta) and ChromaDB are cited as common open-source implementations.
  • LoRA (Low-Rank Adaptation): Parameter-efficient fine-tuning method that trains only a small number of additional low-rank matrices rather than all model weights, dramatically reducing compute and memory requirements.
  • QLoRA (Quantized LoRA): Extends LoRA by also quantizing the base model weights (e.g. to 4-bit), enabling fine-tuning of very large models on limited GPU hardware.

Key Claims and Findings

  • RAG and fine-tuning are complementary, not mutually exclusive — the appropriate choice depends on whether the primary need is dynamic knowledge access (RAG) or consistent domain-specific behaviour (fine-tuning).
  • Fine-tuning offers lower inference latency (knowledge is in weights; no retrieval step), while RAG has lower upfront compute cost but higher per-query resource consumption.
  • Both methods reduce hallucination relative to a baseline LLM, though neither eliminates it.
  • The practical heuristic: prefer RAG when knowledge is too extensive or frequently changing to embed in model weights; prefer fine-tuning when stable, consistent domain behavior and response style are required.
  • LoRA and QLoRA make fine-tuning accessible for large models (e.g. LLaMA 65B) without requiring full-scale GPU clusters.

Terminology

  • RAG pipeline: The sequence — query embedding → ANN search in vector DB → document retrieval → context injection into prompt → LLM generation.
  • Vector database: A specialised data store that indexes document embeddings and supports efficient similarity search. FAISS targets high-performance large-scale search; ChromaDB targets small-to-medium tasks.
  • JSONL fine-tuning format: OpenAI’s standard data format for supervised fine-tuning — one JSON object per line, each containing a messages array with system, user, and assistant roles.
  • RAFT: Retrieval Augmented Fine-Tuning — fine-tune for domain knowledge first, then layer RAG for dynamic retrieval.

Connections to Existing Wiki Pages

  • index — Part 4 operationalises a RAG pipeline (NVIDIARerank, message accumulation, parallel search) that directly applies the retrieval mechanics described here.
  • index — The DLI course covers RAG and structured output in the context of agentic systems; this article provides a standalone reference on the RAG vs. fine-tuning decision.
  • Generative AI LLM Exam Study Guide — The NCA-GENM study guide covers RAG as part of the generative AI production lifecycle, complementing this article’s implementation-level treatment.
  • index — Part 2 covers data flywheel and fine-tuning patterns in the agentic AI context; this article provides the foundational RAG vs. fine-tuning framing.
  • DeepSeek-R1 Incentivizing Reasoning Capability in LLMs via Reinforcement Learing — DeepSeek is cited here as an open-source base model candidate for fine-tuning.

Graph View

Backlinks