Statistics glossary

Clear, one-line definitions of the Statistics terms used across the OgbonLab textbooks. Each entry links to the interactive sections where the idea is taught.

aic

Akaike Information Criterion: −2·log L + 2k; estimates out-of-sample predictive accuracy via Kullback-Leibler divergence.

alternative hypothesis

H₁ (or Hₐ): the rival claim accepted when the data give sufficient evidence against H₀.

asymptotic normality

√n(θ̂ − θ) ⇒ N(0, V) as n → ∞ for a wide class of estimators (notably MLEs under regularity).

bca

Bias-corrected and accelerated bootstrap: improves percentile-interval coverage by correcting for bias and skewness.

benjamini-hochberg

A stepwise procedure that controls the FDR at level q by comparing ordered p-values to (i/m)·q.

biased

An estimator θ̂ is biased when its expected value differs from the parameter it estimates: E[θ̂] ≠ θ.

See: Why your sample histogram is biased

bonferroni

A simple FWER correction that uses α/m as the per-test significance level when testing m hypotheses.

bootstrap

Resampling with replacement from the observed sample to approximate the sampling distribution of a statistic.

See: Bootstrap CIs (percentile, BCa, basic), Bootstrap, deeper: parametric vs nonparametric

breakdown point

The maximum fraction of arbitrarily bad observations an estimator can tolerate before it gives an arbitrarily wrong answer; the median's breakdown point is 50%.

chi-squared test

A test using a χ² statistic for goodness-of-fit or independence in contingency tables.

confidence interval

An interval estimate [L, U] for a parameter such that, over repeated sampling, the interval covers the true parameter with a specified confidence level (e.g. 95%).

See: Prediction intervals vs confidence intervals

consistency

An estimator is consistent if θ̂ₙ →ᵖ θ as n → ∞; precision improves with more data.

See: Consistency, asymptotics, and "large enough"

coverage

The long-run proportion of confidence intervals that contain the true parameter; ideally equals 1 − α.

See: Binning QC and coverage maps

cramer-rao bound

Var(θ̂) ≥ 1/I(θ) for unbiased θ̂; sets the minimum achievable variance and defines efficiency.

delta method

If √n(θ̂ − θ) ⇒ N(0, V) and g is differentiable at θ, then √n(g(θ̂) − g(θ)) ⇒ N(0, g'(θ)²·V).

distribution-free

Describing a statistical method that makes no parametric assumption about the underlying distribution, such as the empirical CDF or rank tests.

See: Quantile regression and distribution-free CIs

effect size

A scale-free measure (e.g., Cohen's d, Pearson r) of the magnitude of a difference or relationship.

See: Type-I, Type-II, power, and effect size

efficiency

Among unbiased estimators, the one with smallest variance; the Cramér-Rao bound sets the asymptotic floor.

See: Sweep Efficiency, Bias-variance, MSE, and the efficiency frontier

empirical

Based on observed data rather than theoretical derivation; the empirical distribution puts mass 1/n at each observed sample.

empirical distribution

F̂_n(x) = (1/n) Σ 1{Xᵢ ≤ x}; the CDF that places mass 1/n on each observation.

estimator

A function θ̂(X₁,...,Xₙ) of the sample used to infer an unknown parameter θ from data.

See: Robust and M-estimators, Robust variogram estimators

exploratory data analysis

Tukey's data-centric approach using graphics and summaries to understand structure before formal modelling.

See: Exploratory data analysis for spatial data

fdr

False discovery rate: the expected proportion of false positives among rejected hypotheses; less conservative than FWER.

fisher information

I(θ) = E[(∂ log f(X;θ)/∂θ)²]; measures how much the data tells you about θ; inverse bounds estimator variance.

See: Fisher information and the Cramér-Rao bound

fwer

Family-wise error rate: the probability of making at least one Type I error across a family of tests.

holm correction

A stepwise FWER correction more powerful than Bonferroni; uses α/(m − i + 1) at the i-th ordered p-value.

inadmissible

An estimator is inadmissible when another estimator has lower risk than it for every parameter value and strictly lower for some.

iqr

Interquartile range: Q₃ − Q₁; a robust measure of spread used in boxplots and outlier rules.

k-fold cv

Partition the data into k folds; for each fold, fit on the others and evaluate on the held-out fold; average errors.

kernel density estimate

f̂(x) = (1/(nh)) Σ K((x − Xᵢ)/h); a smooth nonparametric density estimate with bandwidth h and kernel K.

kolmogorov-smirnov test

A non-parametric test based on the supremum distance between empirical and reference CDFs.

likelihood

The probability (or density) of the observed data viewed as a function of the unknown parameter θ.

See: Maximum likelihood, Profile-likelihood and likelihood-ratio CIs

likelihood ratio test

Compares 2·log(L(θ̂)/L(θ̂₀)) to a χ² distribution; nested-model test with broad applicability.

loocv

Leave-one-out cross-validation: k-fold CV with k = n; nearly unbiased but high-variance and expensive.

mann-whitney test

A non-parametric two-sample test based on ranks; assesses whether one distribution is stochastically larger.

margin of error

Half the width of a confidence interval; the maximum likely deviation of the sample estimate from the true parameter at a chosen confidence level.

mean

The arithmetic average X̄ = (1/n) Σ xᵢ; sensitive to outliers but minimises sum of squared deviations.

median

The middle value of an ordered sample; the 0.5 quantile, robust to outliers and skew.

method of moments

An estimator that equates sample moments to population moments and solves for the parameters.

See: Method of moments

mle

Maximum likelihood estimator: the value θ̂ that maximises the likelihood L(θ; data) of the observed sample.

null hypothesis

H₀: the default claim being tested, often a 'no effect' or 'parameter equals a specific value' statement.

observed fisher information

The negative Hessian of the log-likelihood evaluated at θ̂; a sample-based plug-in for I(θ).

one-sample t-test

Tests H₀: μ = μ₀ using t = (X̄ − μ₀)/(s/√n) ∼ t(n − 1) under H₀ for normal data.

outlier

An observation far from the bulk of the data, often defined via z-scores or IQR rules; may signal error or genuine tail behaviour.

p-value

Under H₀, the probability of observing data at least as extreme as the sample; small values weigh against H₀.

See: What a p-value is, and what it is not

paired t-test

A t-test on the within-pair differences of matched observations; controls for between-pair variation.

parametric bootstrap

Bootstrap that samples from a fitted parametric model rather than the empirical distribution.

percentile

A quantile expressed on a 0-100 scale; the 90th percentile is the value below which 90% of the data lie.

See: Bootstrap CIs (percentile, BCa, basic)

percentile bootstrap

A confidence interval formed from the α/2 and 1 − α/2 quantiles of the bootstrap distribution of θ̂.

permutation test

A non-parametric test that computes the null distribution by repeatedly relabelling group assignments.

See: Permutation tests and exchangeability

pivotal quantity

A function of the data and θ whose distribution does not depend on θ; used to construct exact confidence intervals.

plug-in principle

Estimate a functional T(F) by T(F̂_n) where F̂_n is the empirical CDF; the philosophical basis of the bootstrap.

power

1 − β, the probability of correctly rejecting H₀ when a specific alternative is true; rises with sample size and effect size.

quantile

The value q_τ such that P(X ≤ q_τ) = τ; the median is q_{0.5}, quartiles are q_{0.25}, q_{0.5}, q_{0.75}.

See: Histograms, CDFs, and quantile-quantile plots, Quantile regression and distribution-free CIs

sampling distribution

The distribution of a statistic θ̂(X₁,...,Xₙ) across hypothetical repeated samples from the population.

See: Sampling distributions and the standard error

score function

U(θ) = ∂ log L(θ)/∂θ; has mean 0 at the true θ and variance equal to the Fisher information.

score test

Rao's test based on the score evaluated at θ₀; useful when the MLE is hard to compute under H₁.

significance level

α, the pre-specified maximum tolerated Type I error rate, typically 0.05 or 0.01.

standard deviation

The square root of variance, σ = √Var(X); a measure of dispersion with the same units as X.

standard error

The standard deviation of a sampling distribution, typically SE = σ/√n for the sample mean.

See: Sampling distributions and the standard error

statistical inference

The branch of statistics concerned with drawing conclusions about a population or model parameter from sample data, with quantified uncertainty.

sufficient statistic

A statistic T(X) is sufficient for θ if the conditional distribution of X given T does not depend on θ.

t-test

A test for the mean (or difference of means) using a t-statistic; reduces to a normal test when σ is known.

See: t-tests, χ², F-tests done by hand

two-sample t-test

Compares the means of two independent samples; uses pooled or Welch standard error depending on equal-variance assumption.

type i error

Rejecting H₀ when it is in fact true; its probability is the significance level α.

type ii error

Failing to reject H₀ when H₁ is in fact true; its probability is β, and 1 − β is the test's power.

unbiased

An estimator θ̂ is unbiased if E[θ̂] = θ for every value of the parameter; bias = E[θ̂] − θ.

wald ci

θ̂ ± z·SE(θ̂); a Wald-style interval based on asymptotic normality of θ̂.

wald test

Uses (θ̂ − θ₀)/SE(θ̂) ⇒ N(0, 1) for large n; convenient but less reliable than LRT in small samples.

wilcoxon test

A non-parametric rank-based test for the median or for paired differences; robust to outliers.