RNNs and LSTMs

Recurrent networks read a sequence one step at a time, carrying a memory (hidden state) forward.

• Good for ordered data: sensor readings, text, prices.
• LSTMs use gates to remember information over long sequences.
• Steps run one after another, so they're slow to train on long text.

state = 0.0
for x in [1.0, 2.0, 3.0]:
    state = 0.5 * state + x      # memory of earlier steps fades
print(state)

4.25

Remember: Transformers have mostly replaced RNNs for language.

Common mistake: Shuffling time steps, which destroys the order the model depends on.