Retrieval-augmented generation (RAG)

Embeddings

An embedding turns text into a vector of numbers, so texts with similar meaning end up close together.

• 'Reset my password' and 'forgot login' get nearby vectors.
• Embed questions and documents with the same model.
• Normalised vectors make the dot product equal cosine similarity.

import numpy as np
v = np.array([3.0, 4.0])
print((v / np.linalg.norm(v)).tolist())

[0.6, 0.8]

Remember: Same model for questions and documents, always.

Common mistake: Switching embedding models without re-embedding the old documents.

Similarity search

Similarity search finds the chunks whose vectors are closest to the question's vector.

• Score with cosine similarity or dot product.
• Big collections use approximate search (like HNSW) for speed.
• Hybrid search adds keyword search (BM25) for exact terms like codes.

import numpy as np
q = np.array([1.0, 0.0])
docs = {"a": np.array([0.9, 0.1]), "b": np.array([0.1, 0.9])}
print(max(docs, key=lambda k: float(q @ docs[k])))

a

Remember: Meaning → vectors. Exact strings → keywords. Often both.

Common mistake: Expecting vector search alone to find an exact product code like SKU-48213.

Chunking

Chunking splits documents into pieces small enough to match a question precisely and fit in the prompt.

• A few hundred tokens per chunk is common.
• Overlap (say 100 tokens) keeps ideas that cross a boundary whole.
• Split on headings and paragraphs, not mid-sentence.

doc, size, overlap = 10_000, 500, 100
step = size - overlap
print(-(-(doc - size) // step) + 1)

25

Remember: Chunk size is a setting to test, not a fixed rule.

Common mistake: Embedding a whole 50-page PDF as one vector, which blurs every topic together.

Vector databases

A vector database stores each chunk's embedding with its text and metadata, and searches them by similarity.

• Examples: pgvector (in Postgres), Qdrant, Pinecone, Chroma.
• Metadata (source, date, team) lets you filter and cite.
• Re-index documents when they change.

dim, chunks, bytes_per_number = 768, 100_000, 4
print(round(dim * chunks * bytes_per_number / 1_000_000, 1), "MB")

307.2 MB

Remember: Store the source with every chunk, so answers can cite it.

Common mistake: Leaving old chunks in the index after a policy changes.

Retrieval and ranking

RAG retrieves the most relevant chunks for a question, then the LLM answers from them.

• Steps: embed the question → search → pick the top k → prompt the LLM.
• A re-ranker re-scores the top 50 more carefully and keeps the best 5.
• More chunks isn't always better: irrelevant text distracts the model.

scores = {"c1": 0.82, "c2": 0.91, "c3": 0.40, "c4": 0.77}
print(sorted(scores, key=scores.get, reverse=True)[:2])

['c2', 'c1']

Remember: Good retrieval matters more than a clever prompt.

Common mistake: Stuffing 100 chunks into the prompt and getting a vaguer answer.

Grounded answers

A grounded answer says only what the retrieved sources support, and shows where each claim came from.

• Tell the model to answer only from the provided context.
• Ask it to cite sources, like [HR Policy, p.4].
• If the answer isn't in the context, it should say so.

Example: Context says leave is 24 days. Question: 'How many leave days?' Answer: '24 days [Leave Policy, section 2].'

Remember: Retrieved text is data, never instructions.

Common mistake: Letting a retrieved page that says 'ignore your instructions' change the model's behaviour.

Retrieval metrics

Retrieval metrics check whether the right chunks were found, before you judge the answer.

• Recall@k: share of the relevant chunks found in the top k.
• Precision@k: share of the top k that are relevant.
• MRR rewards putting the first relevant chunk near the top.

relevant, found_in_top5 = 4, 3
print(f"recall@5 {found_in_top5 / relevant:.0%}, precision@5 {found_in_top5 / 5:.0%}")

recall@5 75%, precision@5 60%

Remember: If retrieval misses, no prompt can fix the answer.

Common mistake: Only scoring final answers, so you can't tell whether search or generation failed.

Answer quality

Answer quality checks two things: is it faithful to the sources, and does it answer the question asked?

• Faithfulness: every claim is supported by the retrieved context.
• Relevance: it addresses what the user actually asked.
• Build a golden set of questions, answers and source passages.

Example: Question about leave carry-over; answer correctly quotes the leave count but never mentions carry-over → faithful, not relevant.

Remember: Check both: faithful but off-topic is still a bad answer.

Common mistake: Trusting automatic scores (like RAGAS) without spot-checking some by hand.