Lesson 3 of 4

Transformers

Watch, read the card, then practise. Self-attention · BERT and GPT

Video coming soon

The video for this lesson is being recorded.

Meanwhile, the notes below and the 60-second cards cover the same ideas.

Self-attention
In self-attention, every token looks at every other token to decide what's relevant to it.
Remember: Multi-head attention looks for several kinds of relationships at once.
BERT and GPT
BERT reads text in both directions to understand it; GPT reads left to right to generate it.
Remember: Understand or classify → encoder. Write or answer → decoder.

Try the example in Code Lab

tokens = 512
print(tokens * tokens)

Lesson 3 of 4

Watch, read the card, then practise. Self-attention · BERT and GPT

Video coming soon

The video for this lesson is being recorded.

Meanwhile, the notes below and the 60-second cards cover the same ideas.

Self-attention
In self-attention, every token looks at every other token to decide what's relevant to it.
Remember: Multi-head attention looks for several kinds of relationships at once.
BERT and GPT
BERT reads text in both directions to understand it; GPT reads left to right to generate it.
Remember: Understand or classify → encoder. Write or answer → decoder.

tokens = 512
print(tokens * tokens)