← Deep learning
Lesson 4 of 5
Sequence models
Watch, read the card, then practise. RNNs and LSTMs · Attention
The video for this lesson is being recorded.
Meanwhile, the notes below and the 60-second cards cover the same ideas.
Practise this topicRNNs and LSTMs
Recurrent networks read a sequence one step at a time, carrying a memory (hidden state) forward.
Remember: Transformers have mostly replaced RNNs for language.
Attention
Attention lets each position look at every other position and take a weighted mix of the useful ones.
Remember: Attention = relevance scores → softmax → weighted average.
Try the example in Code Lab
state = 0.0
for x in [1.0, 2.0, 3.0]:
state = 0.5 * state + x # memory of earlier steps fades
print(state)