Generative AI and LLMs

Tokens and context

LLMs read and write tokens, and can only see a limited number at once: the context window.

• One English token is about ¾ of a word.
• The prompt and the reply share the context window.
• APIs charge per token, often more for output tokens.

window, prompt = 8000, 6500
print(window - prompt, round(750 / 0.75))

1500 1000

Remember: Long documents must fit in the window, or be retrieved in pieces.

Common mistake: Filling the window with documents and leaving no room for the answer.

Sampling and temperature

An LLM picks each next token by sampling from probabilities. Temperature controls how adventurous that is.

• Low temperature (0 to 0.3): predictable, best for facts and code.
• High temperature (0.8+): varied, good for brainstorming.
• Top-p keeps only the likeliest tokens that add up to p.

import numpy as np
logits = np.array([2.0, 1.0])
for t in (1.0, 0.5):
    p = np.exp(logits / t) / np.exp(logits / t).sum()
    print(t, round(float(p[0]), 2))

1.0 0.73
0.5 0.88

Remember: Facts and formats → low temperature.

Common mistake: Using a high temperature for data extraction and getting different answers each time.

Prompt design

A good prompt says the task, the context, the format and who the answer is for.

• Be specific: 'Summarise in 3 bullets for a first-year student.'
• Use a system prompt for the role and rules.
• Ask for structured output (like JSON with named fields) when code will read it.

Example: Weak: 'Explain SQL joins.' Better: 'Explain INNER and LEFT JOIN to a beginner in 5 sentences, with a students/departments example.'

Remember: If a person would need more context, so does the model.

Common mistake: One-word prompts like 'Summarise.' and then blaming the model.

Examples and step-by-step

Few-shot prompting shows a few worked examples first; asking for steps helps on multi-step problems.

• Zero-shot: instructions only. Few-shot: a few examples.
• Examples set the format and style you want.
• 'Think step by step' helps with maths and logic.

Example: Review: 'Fast delivery' → positive. Review: 'Broke in a week' → negative. Review: 'Okay for the price' → ?

Remember: Use varied, balanced examples.

Common mistake: Giving examples that all have the same answer, so the model leans towards it.

Fine-tuning

Fine-tuning trains a pre-trained model further on your own examples, so a style or skill becomes built in.

• Needs input → ideal output pairs, often hundreds to thousands.
• LoRA trains small added matrices, so it's cheap.
• Too narrow training can make the model forget general skills.

Example: 1,000 support tickets paired with good replies → a model that answers in your company's tone.

Remember: Fine-tune behaviour and format, not fast-changing facts.

Common mistake: Fine-tuning to teach facts that change every week.

Prompting or fine-tuning

Try prompting first, add retrieval for facts, and fine-tune only when those can't reach the quality you need.

• Prompting: instant to change, no training.
• RAG: for knowledge that changes or is private.
• Fine-tuning: for consistent style or format at scale, or a smaller, cheaper model.

Example: Prices change weekly → RAG. Every reply must follow a strict tone and format → consider fine-tuning.

Remember: Prompt → RAG → fine-tune, in that order.

Common mistake: Jumping to fine-tuning before writing a good prompt and test set.

Hallucinations

A hallucination is a confident answer that is wrong or made up.

• LLMs predict likely text, which can include invented facts or citations.
• Give the model source text and tell it to answer only from it.
• Let it say 'I don't know'.

Example: Ask about a paper that doesn't exist and a model may describe it in detail. Grounding and checking prevent this.

Remember: Verify facts, numbers and citations before trusting them.

Common mistake: Pasting an LLM's citations into a report without checking they exist.

Evaluating outputs

To improve an LLM app, measure it: build a fixed test set and score every change against it.

• Exact match for short answers; rubrics for longer ones.
• LLM-as-a-judge scales grading; spot-check it by hand.
• Normalise answers (case, spaces) before comparing.

answers = ["Delhi", "mumbai ", "Chennai"]
expected = ["delhi", "Mumbai", "Kolkata"]
norm = lambda s: s.strip().lower()
print(sum(norm(a) == norm(e) for a, e in zip(answers, expected)), "of", len(expected))

2 of 3

Remember: No test set, no way to know if a change helped.

Common mistake: Changing the prompt to fix one case and silently breaking others.