Self-attention

In self-attention, every token looks at every other token to decide what's relevant to it.

• Each token makes a query, a key and a value.
• Weights = softmax(Q·Kᵀ ÷ √d); output = weighted mix of values.
• Cost grows with length squared: 512 tokens → 262,144 scores per head.

tokens = 512
print(tokens * tokens)

262144

Remember: Multi-head attention looks for several kinds of relationships at once.

Common mistake: Forgetting positional encodings: attention alone ignores word order.