Retrieval-Augmented Generation (RAG): How AI Learns to Cite Its Sources

Large language models are great at reasoning and writing, but they have two hard limits: their knowledge is frozen at training time, and they can't see your private data. Retrieval-Augmented Generation (RAG) fixes both by giving the model a way to look things up before it answers.

The core idea

Instead of asking a model a question directly and hoping it "remembers" the right answer, a RAG pipeline:

1. Retrieves relevant chunks of text from a knowledge base (docs, PDFs, a database, your codebase)
2. Augments the prompt with those chunks as context
3. Generates an answer grounded in that context, ideally with citations

User question
   │
   ▼
Embed question ──▶ Vector search over knowledge base
   │                          │
   │                          ▼
   │                 Top-k relevant chunks
   │                          │
   └──────────────┬───────────┘
                   ▼
        Prompt = question + chunks
                   ▼
              LLM generates answer

Why not just fine-tune?

Fine-tuning bakes knowledge into model weights — it's slow, expensive, and hard to update. RAG keeps the model frozen and swaps out the context instead, so:

• Updating knowledge is as simple as re-indexing documents, no retraining
• You can cite exactly which source backed an answer
• It works with any off-the-shelf model, including hosted APIs

A minimal retrieval pipeline

import { OpenAI } from "openai";
 
const openai = new OpenAI();
 
async function embed(text) {
  const res = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
  });
  return res.data[0].embedding;
}
 
async function retrieve(query, vectorStore, topK = 4) {
  const queryVector = await embed(query);
  return vectorStore.similaritySearch(queryVector, topK);
}
 
async function answer(query, vectorStore) {
  const chunks = await retrieve(query, vectorStore);
  const context = chunks.map((c) => c.text).join("\n---\n");
 
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content:
          "Answer using only the provided context. Cite sources by chunk index.",
      },
      { role: "user", content: `Context:\n${context}\n\nQuestion: ${query}` },
    ],
  });
 
  return response.choices[0].message.content;
}

Things that actually matter in practice

• Chunking strategy — split by semantic boundaries (headings, paragraphs), not fixed character counts. Bad chunking is the #1 cause of bad retrieval.
• Reranking — a fast vector search gets you candidates; a reranker (cross-encoder) picks the best few before they hit the prompt. Big quality jump for cheap.
• Metadata filtering — tag chunks with source, date, or type so you can narrow the search space before doing similarity search.
• Evaluation — track retrieval precision separately from generation quality. If the model gives a wrong answer, first check whether the right chunk was even retrieved.

Where this shows up in real projects

I've used a similar pattern outside of pure RAG chat — the Puppeteer pipeline I built at Cygnus Analytics that digitized 500+ Sanskrit manuscript pages relies on the same principle: extract structured content once, store it efficiently, then let downstream tools (search, LLMs, translation) query it instead of re-processing raw scans every time. RAG is really just this idea applied to unstructured text at LLM-query time.

If you're building anything that needs an LLM to reason over your own data — internal docs, support tickets, codebases — RAG is usually the right first tool to reach for before fine-tuning or building anything custom.