Deep Learning

92 flashcards by MentisLibrary

How neural networks are built and trained: activations, loss functions, backpropagation and optimisers, regularisation and normalisation, convolutional and recurrent layers, transformers, generative models, and the practical habits of a training run.

What makes a neural network deep?

Stacking several hidden layers, so each layer transforms the representation produced by the one before it.

What does a single artificial neuron compute?

A weighted sum of its inputs plus a bias, passed through a nonlinear activation function.

Why does a network without activation functions gain nothing from depth?

Because the composition of linear maps is itself a linear map, so any number of linear layers is equivalent to one.

What is a fully connected layer?

A layer where every input is connected to every output by its own weight, also called a dense or linear layer.

What is the difference between a parameter and a hyperparameter?

Parameters are learned by training, such as weights and biases. Hyperparameters are set before training, such as learning rate, depth and batch size.

What happens in the forward pass?

Input flows layer by layer to the output, and the loss is computed by comparing that output with the target.

What does the universal approximation theorem say?

A feedforward network with one hidden layer and enough units can approximate any continuous function on a compact domain to arbitrary accuracy.

What is a tensor in deep learning?

A multidimensional array, the data structure every layer consumes and produces.

What are logits?

The raw, unnormalised scores a network outputs before a sigmoid or softmax turns them into probabilities.

What does the sigmoid function do?

Squashes any real number into the range 0 to 1, so it can be read as a probability.

How does tanh differ from sigmoid?

It has the same S shape but outputs from -1 to 1, so it is zero centred.

What is ReLU and why is it the default hidden activation?

It passes positive values through unchanged and outputs zero otherwise. It is cheap and does not saturate for positive inputs, so gradients flow.

What is the dying ReLU problem?

A unit whose input is always negative outputs zero and receives zero gradient, so it can never recover.

What does leaky ReLU change?

It gives negative inputs a small nonzero slope so the gradient never vanishes completely.

What is softmax used for?

Turning a vector of logits into a probability distribution that sums to 1, for multi-class classification.

Why do transformer models tend to use GELU rather than ReLU?

GELU is a smooth activation that scales an input by how likely it is to be kept under a Gaussian, and it consistently trains slightly better in these architectures.

What is a loss function?

A single number measuring how wrong the model's outputs are, which training tries to minimise.

When do you use cross-entropy loss?

For classification, where it measures the negative log probability the model assigned to the correct class.

When is mean squared error the right loss?

For regression, where the target is a continuous value and large errors should be penalised disproportionately.

What does gradient descent do?

Moves every parameter a small step in the direction that most reduces the loss, which is opposite to the gradient.

What does backpropagation actually compute?

The gradient of the loss with respect to every parameter, by applying the chain rule backwards through the computation graph.

What is stochastic gradient descent?

Gradient descent that estimates the gradient from a small batch of examples rather than the whole dataset.

What do epoch, batch and iteration mean?

An epoch is one pass over the training set, a batch is the group of examples in one update, and an iteration is one update.

What does the learning rate control?

The size of each parameter update, and it is the hyperparameter that most often decides whether training works at all.

What does momentum add to gradient descent?

It accumulates a running velocity of past gradients, so consistent directions build speed and oscillating ones cancel out.

What does Adam combine?

Momentum on the gradient with a per-parameter step size scaled by a running estimate of the gradient's squared magnitude, plus bias correction for the early steps.

Why decay the learning rate during training?

Large steps make fast early progress, while small later steps let the model settle into a minimum instead of bouncing around it.

What is learning rate warmup?

Starting at a very small learning rate and raising it over the first few hundred or few thousand steps.

What is gradient clipping?

Rescaling the gradient when its norm exceeds a threshold, so a single huge gradient cannot destroy the weights.

What are exploding and vanishing gradients?

Repeated multiplication through many layers makes gradients grow without bound or shrink towards zero, so early layers get useless updates.