Machine Learning glossary

Clear, one-line definitions of the Machine Learning terms used across the OgbonLab textbooks. Each entry links to the interactive sections where the idea is taught.

62 terms

accuracy: (TP + TN) / total; the fraction of correct predictions, misleading on imbalanced classes.; See: The cost-vs-accuracy front, Accuracy plots and reliability diagnostics
anomaly detection: Identifying samples that deviate from the bulk of the data, often via density estimation, isolation forest, or autoencoder reconstruction error.
auc: Area Under the ROC curve; equals the probability that a random positive scores higher than a random negative under the model.
bias-variance tradeoff: Decomposition of expected error into bias (systematic miss) plus variance (sensitivity to data) plus irreducible noise; complex models trade bias for variance.
classification: Supervised task in which the target y is a discrete class label and the model outputs a category or class-probability vector.; See: Multiple classification, Attribute combination: RGB blending and classification
clustering: Unsupervised grouping of samples into clusters by similarity, without using any class labels.
confusion matrix: A K×K table whose (i, j) entry counts samples of true class i predicted as class j; the basis of most classification metrics.
cost function: An alternative name for the loss being minimised during model training, especially in classical machine learning.
data augmentation: Synthesising extra training examples from existing ones via label-preserving transforms (flips, noise, time shifts); reduces overfitting.
data leakage: Inadvertent inclusion of information about the target in the training features (e.g. future data, near-duplicate splits) producing overly optimistic validation scores.
dimensionality reduction: Any technique (PCA, t-SNE, UMAP, autoencoder) that maps high-dimensional features to a lower-dimensional space while preserving structure.; See: Dimensionality Reduction, Dimensionality Reduction Practice
domain shift: A mismatch between training and deployment data distributions that degrades model performance; a central problem in transfer learning.
early stopping: Halting training when validation error stops improving for a chosen patience; an implicit regulariser that prevents overfitting to the training set.
embedding: A learned dense vector representation of a discrete or high-dimensional object, placed in a metric space where geometry reflects semantics.; See: Fourier feature embeddings
f1 score: Harmonic mean of precision and recall, F1 = 2·P·R / (P + R); balances the two and penalises extreme imbalance between them.
feature: A measured or engineered input variable supplied to a machine-learning model (e.g. a well-log channel, a seismic attribute).; See: Feature Engineering, Fourier feature embeddings
feature scaling: Standardising features to comparable ranges (z-score, min-max) so that distance-based and gradient-based learners are not dominated by units.
feature selection: Choosing a subset of input variables that contributes most to predictive performance; reduces overfitting and improves interpretability.
fine-tuning: Resuming training of a pretrained model on the target dataset, usually with a small learning rate, to specialise it to the new task.
gradient boosting: An ensemble method that sequentially fits each new weak learner to the gradient of the loss with respect to the previous ensemble's predictions.; See: Trees, random forests, and gradient boosting
hallucination: An ML model output that is fluent and confident but factually wrong or unsupported by the input; common in generative models.
interpretable: A model whose internal decision logic a human can inspect and follow, such as a shallow tree or a sparse linear model.
k-fold cross-validation: Validation scheme that splits data into k folds, training on k−1 and validating on the remaining fold, rotated k times for a more stable performance estimate.
k-means: Clustering algorithm that alternates assigning each point to the nearest of k centroids with updating centroids as the within-cluster means, minimising within-cluster variance.
l1 regularization: Adding λ·Σ|wᵢ| to the loss; encourages sparse weights and acts as a feature selector.
l2 regularization: Adding λ·Σwᵢ² to the loss; shrinks weights toward zero (weight decay) without producing exact zeros.
label: The target value associated with a training sample; the quantity a supervised learner is trained to predict.
labeled: Describing data points that come paired with their target value y, enabling supervised learning of the input-to-output map.
latent space: The low-dimensional space in which an autoencoder, VAE, or generative model represents inputs; geometry there exposes structure invisible in raw features.
linear regression: A regression model ŷ = wᵀx + b fit by minimising squared error; closed-form solution given by the normal equations.
linearly separable: A classification problem in which the classes can be perfectly separated by a single hyperplane in the feature space.
logistic regression: A linear classifier modelling P(y = 1 | x) = σ(wᵀx + b); fit by minimising binary cross-entropy.; See: Linear and Logistic Regression, Logistic regression and odds ratios
max depth: A regularisation hyperparameter capping the maximum number of splits from root to leaf in a decision tree or random-forest base learner.
naive bayes: A probabilistic classifier that applies Bayes' rule under the (often unrealistic) assumption that features are conditionally independent given the class.; See: Naive Bayes, Naive Bayes Practice
overfitting: Fitting noise in the training data so that training error keeps falling while validation error rises; high variance, low bias.; See: Overfitting, Underfitting, Variance, and Bias
pca: Common abbreviation for Principal Component Analysis; the dominant linear dimensionality-reduction technique.
permutation importance: A feature-importance measure obtained by randomly shuffling a feature's values and observing the drop in validation performance.
precision: TP / (TP + FP); the fraction of positive predictions that are actually positive.
principal component analysis: An orthogonal linear projection onto the directions of maximum variance in the data; obtained from the eigendecomposition of the covariance matrix.
random forest: An ensemble of decision trees grown on bootstrap samples with random feature subsets; predictions are averaged (regression) or voted (classification).; See: Random Forest, Random Forest Practice
recall: TP / (TP + FN); the fraction of true positives that were retrieved (also called sensitivity).
regression: Supervised task in which the target y is a continuous real-valued quantity and the model minimises a numerical-error loss.; See: Robust regression, Regression discontinuity
regularization: Any technique that constrains a model to reduce overfitting, including weight penalties (L1/L2), dropout, early stopping, and data augmentation.
roc curve: Receiver Operating Characteristic: plot of true-positive rate versus false-positive rate as the decision threshold varies.
self-supervised learning: Training on labels manufactured from the inputs themselves (masking, contrastive views) so that no human annotation is required.
stochastic: Involving randomness; in optimisation, refers to using random subsamples (mini-batches) so that each gradient estimate is a noisy approximation of the true gradient.; See: Stochastic Heterogeneity
supervised classification: Learning a function from labeled examples (x, y) that predicts a discrete class label y for new inputs x.
supervised learning: Learning from a labelled dataset {(xᵢ, yᵢ)} by fitting a function f_θ(x) ≈ y; produces classifiers or regressors.
support vector machine: A margin-maximising classifier that finds the hyperplane (or kernel-induced surface) separating classes with the largest possible margin to support vectors.
t-sne: t-distributed Stochastic Neighbour Embedding: a non-linear visualisation method that preserves local neighbour structure in a 2D or 3D embedding.
test set: The data partition held out until model selection is complete and used once to estimate generalisation performance.
training loss: The scalar value of the objective function on the training set at a given optimisation step; what gradient descent is actively minimising.
training set: The data partition used to fit a model's parameters; performance on it alone is a poor indicator of generalisation.
transfer learning: Reusing a model trained on a large source dataset as the starting point for a smaller target task; common with CNNs and pretrained encoders.
umap: Uniform Manifold Approximation and Projection: a topology-based non-linear embedding that preserves both local and some global structure; faster than t-SNE.
uncertainty flagging: Marking model predictions whose calibrated confidence falls below a threshold so a human reviewer can vet them before they are used.
underfitting: Failure to capture relevant structure, resulting in high training and validation error; high bias, low variance.; See: Overfitting, Underfitting, Variance, and Bias
unlabeled: Describing data points with inputs only and no target value, used in unsupervised or self-supervised learning.
unsupervised clustering: Discovering natural groupings in unlabeled data by maximising within-cluster similarity and between-cluster separation.
unsupervised learning: Learning structure (clusters, density, low-dimensional embeddings) from unlabelled data {xᵢ}.
validation set: The data partition used during training to tune hyperparameters and choose model architectures, kept disjoint from the test set.
xgboost: A scalable gradient-boosted-tree library widely used in tabular ML; features regularised objectives, sparsity-aware splits, and parallel histogram building.