Deep Learning glossary

Clear, one-line definitions of the Deep Learning terms used across the OgbonLab textbooks. Each entry links to the interactive sections where the idea is taught.

activation function

A nonlinear scalar function (ReLU, tanh, sigmoid, …) applied element-wise to a layer's pre-activation, providing the network's nonlinearity.

See: Activation functions and inductive bias

adam

Adaptive Moment Estimation: an optimiser that combines momentum and per-parameter learning-rate scaling using running estimates of the first and second gradient moments.

autoencoder

An unsupervised network that learns to reconstruct its input via a low-dimensional bottleneck (latent code); used for compression, denoising, and feature learning.

backpropagation

The reverse-mode automatic-differentiation algorithm that computes loss gradients with respect to every weight by chain-ruling through the computational graph.

See: The chain rule and backpropagation

batch

A subset of the training data over which loss and gradients are accumulated before a single optimiser step; size influences both noise and throughput.

bias

An additive learnable scalar in a layer's affine map y = Wx + b, allowing the layer to shift its output independently of its inputs.

binary cross-entropy

Two-class cross-entropy L = −(y log ŷ + (1−y) log(1−ŷ)); used with a sigmoid output for binary classifiers.

cnn

Convolutional Neural Network: a deep architecture with weight-shared convolutional filters and pooling, well suited to images, seismic sections, and well-log windows.

computational graph

A directed acyclic graph whose nodes are operations and whose edges are tensors; backpropagation traverses it from outputs to inputs.

See: Auto-differentiation as a computational graph

cross-entropy

Loss L = −Σ yᵢ log(ŷᵢ) used for classification; equals the negative log-likelihood of the data under the predicted categorical distribution.

dropout

A regulariser that randomly zeros each unit's output with probability p during training, approximating an exponentially large ensemble of subnetworks.

epoch

One complete pass of the optimiser through the entire training dataset.

feedforward

A network architecture in which information flows in one direction from inputs through hidden layers to outputs, with no cycles.

gan

Generative Adversarial Network: a generator and discriminator trained in a min-max game so the generator produces samples that look like the training distribution.

gradient descent

Iterative optimiser that updates parameters θ ← θ − η·∇L(θ) using the gradient of the loss; foundation of neural-network training.

See: Gradient descent by hand, Gradient Descent and Sampling Strategies

jax

A NumPy-compatible Python library offering composable function transformations (grad, jit, vmap, pmap) on accelerators; popular for PINNs and scientific ML.

layer

A single transformation x ↦ σ(Wx + b) inside a neural network; layers are stacked to build depth.

learning rate

The scalar step size η that controls how far parameters move along the negative-gradient direction at each optimisation step.

loss function

A scalar L(θ) measuring how badly a model's outputs match the desired targets on the training data; minimised during learning.

See: Loss functions and the loss landscape

lstm

Long Short-Term Memory: a gated RNN cell with input, forget, and output gates that allow long-range dependencies without vanishing gradients.

mse

Mean Squared Error, L = (1/N) Σ (yᵢ − ŷᵢ)²; the default loss for real-valued regression problems.

neural network

A composition of parameterised affine maps and nonlinear activations f(x) = σₙ(Wₙ … σ₁(W₁x + b₁) … + bₙ), trained by minimising a loss.

See: Recurrent Neural Networks, Building a Simple Neural Network

neuron

A single scalar output unit inside a layer, computing σ(wᵀx + b) on its inputs.

See: What is a neuron?, Perceptrons and Neurons: A Simple NN Model

pytorch

A widely used deep-learning framework with eager execution and dynamic computation graphs; the most common platform for PINN research.

relu

Rectified Linear Unit, σ(x) = max(0, x); fast to compute and largely free of vanishing-gradient problems in deep networks.

rnn

Recurrent Neural Network: a network that processes sequences via a hidden state updated at each time step, allowing memory of past inputs.

sgd

Stochastic Gradient Descent: gradient descent in which each step uses the gradient from a single sample or a small mini-batch instead of the full dataset.

sigmoid

Activation σ(x) = 1/(1 + e^(−x)); maps ℝ to (0,1) and is used for binary-classification output layers.

tanh

Activation σ(x) = (e^x − e^(−x))/(e^x + e^(−x)); maps ℝ to (−1, 1) and is the default smooth activation for many PINNs.

tensorflow

A Google deep-learning framework offering both eager and graph-mode execution; the original platform used in the foundational PINN papers.

transformer

A sequence model built on multi-head self-attention and feed-forward blocks; the backbone of modern language and increasingly seismic foundation models.

u-net

An encoder-decoder CNN with skip connections introduced for biomedical segmentation; widely used for seismic fault, salt, and horizon segmentation.

universal approximation theorem

Theorem stating that feedforward networks with a single hidden layer and a non-polynomial activation can approximate any continuous function on a compact set to arbitrary accuracy.

vanishing gradient

Failure mode of deep networks where gradients shrink exponentially with depth, stalling early-layer learning; mitigated by ReLU, residual links, and normalisation.

weight

A learnable scalar entry of a matrix W that multiplies an input feature inside a neural-network layer.

See: Loss-weight sensitivity in FWI-PINN, Gradient pathologies and adaptive weights

weight sharing

Reusing the same parameter at many positions (as in convolution or recurrence); reduces parameter count and imposes a useful symmetry on the model.