Deep learning

One 60-second card per skill, in learning order. 11 of 11 cards ready.

Topic 1

Neural network basics

Neurons and layers · Activation functions

60-second card

Neurons and layers

A neuron multiplies each input by a weight, adds them up with a bias, then applies an activation. Layers are many neurons side by side.

A dense layer is a matrix multiply: outputs = W · x + b.
Parameters per dense layer = inputs × neurons + neurons.
Hidden layers between input and output make a network 'deep'.

inputs, neurons = 4, 3
print(inputs * neurons + neurons)

Prints

Remember: Weights and biases are what training learns.

Common mistake: Forgetting the biases when counting a layer's parameters.

60-second card

Activation functions

Activation functions add curves, so stacked layers can learn more than a straight line.

ReLU: max(0, x). Fast and the usual default for hidden layers.
Sigmoid squashes to 0–1, for yes/no outputs.
Softmax turns scores into probabilities that add to 1, for many classes.

import numpy as np
z = np.array([-2.0, 0.0, 3.0])
print(np.maximum(0, z).tolist())

Prints

[0.0, 0.0, 3.0]

Remember: No activation = the whole network is just one linear model.

Common mistake: Using softmax on a multi-label problem, where each label needs its own sigmoid.

Topic 2

Training networks

Loss functions · Gradient descent · Backpropagation

Open the lesson Practise this topic

60-second card

Loss functions

The loss measures how wrong the predictions are. Training tries to make it smaller.

Regression: mean squared error (MSE) or mean absolute error (MAE).
Classification: cross-entropy, −log(probability of the right class).
Confident wrong answers get a large cross-entropy loss.

import math
print(round(-math.log(0.9), 2), round(-math.log(0.1), 2))

Prints

0.11 2.3

Remember: Number → MSE. Category → cross-entropy.

Common mistake: Using accuracy as the loss. It can't be trained with gradients.

60-second card

Gradient descent

Gradient descent nudges each weight downhill on the loss: weight = weight − learning rate × gradient.

The gradient says which way the loss goes up; you step the other way.
Too large a learning rate overshoots; too small crawls.
Mini-batches: each update uses a small batch, like 32 examples.

w, grad, lr = 2.0, 4.0, 0.1
print(round(w - lr * grad, 2))

Prints

1.6

Remember: If the loss explodes, lower the learning rate first.

Common mistake: Forgetting that one epoch is many steps: examples ÷ batch size.

60-second card

Backpropagation

Backpropagation works out every weight's gradient by passing the error backwards through the layers with the chain rule.

One training step: forward pass, loss, backward pass, update.
The chain rule multiplies derivatives layer by layer.
Tiny derivatives multiplied many times vanish; ReLU and skip links help.

# y = (2x)^2 at x = 3: dy/dx = 2(2x) · 2
x = 3
print(2 * (2 * x) * 2)

Prints

Remember: Forward makes predictions; backward makes gradients.

Common mistake: Thinking backprop is a separate model. It's just how gradients are computed.

Topic 3

Convolutional networks

Convolutions · Pooling

Open the lesson Practise this topic

60-second card

Convolutions

A convolution slides a small filter over an image, computing a weighted sum at each spot to detect a pattern.

Early filters find edges; later ones find shapes and objects.
Output size = (input − filter + 2 × padding) ÷ stride + 1.
The same filter is reused everywhere, so CNNs need few weights.

size, f, pad, stride = 28, 3, 0, 1
print((size - f + 2 * pad) // stride + 1)

Prints

Remember: Filter count = how many patterns a layer looks for.

Common mistake: Forgetting the input channels when counting weights: 3×3 on RGB is 27 per filter.

60-second card

Pooling

Pooling shrinks a feature map, keeping the strongest signals.

2×2 max pooling keeps the largest of every 4 values and halves width and height.
It makes features less sensitive to small shifts.
Global average pooling turns each map into one number before the classifier.

import numpy as np
x = np.array([[1, 3, 2, 0], [2, 4, 1, 5]])
print([int(x[:, i:i + 2].max()) for i in (0, 2)])

Prints

[4, 5]

Remember: Pooling has no weights to learn.

Common mistake: Pooling so much early on that small details are lost.

Topic 4

Sequence models

RNNs and LSTMs · Attention

Open the lesson Practise this topic

60-second card

RNNs and LSTMs

Recurrent networks read a sequence one step at a time, carrying a memory (hidden state) forward.

Good for ordered data: sensor readings, text, prices.
LSTMs use gates to remember information over long sequences.
Steps run one after another, so they're slow to train on long text.

state = 0.0
for x in [1.0, 2.0, 3.0]:
    state = 0.5 * state + x      # memory of earlier steps fades
print(state)

Prints

4.25

Remember: Transformers have mostly replaced RNNs for language.

Common mistake: Shuffling time steps, which destroys the order the model depends on.

60-second card

Attention

Attention lets each position look at every other position and take a weighted mix of the useful ones.

Scores say how relevant each position is; softmax turns them into weights.
Self-attention is the core of the Transformer.
Every position can be processed in parallel, unlike an RNN.

import numpy as np
scores = np.array([2.0, 0.0])
w = np.exp(scores) / np.exp(scores).sum()
print(w.round(2).tolist())

Prints

[0.88, 0.12]

Remember: Attention = relevance scores → softmax → weighted average.

Common mistake: Expecting attention to understand word order alone. Transformers add position information.

Topic 5

Training tips

Dropout and regularisation · Batch norm and learning rate

Open the lesson Practise this topic

60-second card

Dropout and regularisation

Dropout randomly switches off some neurons during training, so the network can't rely on any one of them.

Dropout 0.5 ignores each neuron half the time.
It acts like training many smaller networks and averaging them.
It is turned off when making predictions.

import numpy as np
x = np.array([5.0, 6.0, 7.0, 8.0])
mask = np.array([1, 0, 1, 0])
print((x * mask / 0.5).tolist())   # scaled so the average stays the same

Prints

[10.0, 0.0, 14.0, 0.0]

Remember: Use dropout when the training score is much higher than validation.

Common mistake: Leaving the model in training mode at prediction time, so outputs change every run.

60-second card

Batch norm and learning rate

Batch normalisation keeps each layer's outputs in a steady range, and a good learning-rate schedule makes training fast and stable.

Batch norm rescales outputs to mean 0 and spread 1 per batch, then learns a shift.
Adam adapts each weight's step size and is a common default.
Warm up the learning rate, then lower it over time.

import numpy as np
x = np.array([2.0, 4.0, 6.0])
print(((x - x.mean()) / x.std()).round(2).tolist())

Prints

[-1.22, 0.0, 1.22]

Remember: The learning rate is the first setting to tune.

Common mistake: Using a tiny batch size with batch norm, which makes its averages noisy.